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EXECUTIVE  SUMMARY 


Research  Requirement: 

Advanced  training  in  critical  thinking  (CT)  is  needed  for  adult  populations  in  many  fields 
of  work.  Not  surprisingly  the  United  States  military  is  at  the  forefront  of  the  effort  to  promote 
and  improve  thinking  skills.  Military  leadership  demands  the  application  of  high  quality  CT  for 
effective  battle  command.  Thus,  a  new  Web-based  CT  training  system,  Computerized  Training 
in  Critical  Thinking  (CT)',  was  developed.  The  present  volume  is  the  third  in  a  three  volume 
report  which  describe  a  research  program  in  developing  training  for  critical  thinking.  The  first 
volume  presents  an  overview  of  the  research  effort  that  developed  and  validated  a  theoretical 
model  for  the  training,  selected  and  validated  eight  high  impact  critical  thinking  skills  (CTS)  for 
Army  officers,  and  developed  and  evaluated  a  prototype  training  system.  The  second  volume 
presents,  in  more  detail,  the  model  and  theoretical  basis  for  the  training  and  the  investigations  that 
were  conducted  to  validate  the  model.  This  volume  describes  the  training  that  was  developed, 
including  the  functional  requirements,  pedagogical  principles,  learning  objectives,  organization, 
course  content,  and  evaluation  of  the  training.  The  prototype  training  system  described  in  this 
volume  provides  training  for  two  of  the  eight  identified  CTS.  A  succeeding  report  (Fischer,  Spiker, 
Harris,  McPeters,  &  Riedel,  2008)  describes  the  expanded  training  system  which  trains  all  eight 
CTS. 

Procedure: 

The  goal  of  this  research  was  to  create  easily  accessible,  state-of-the-art  distance-learning 
instruction  that  would  effectively  train  CT.  To  meet  this  objective,  teaching  methods  were 
identified  that  were  likely  to  be  most  effective  at  developing  CT  in  Army  officers.  Two 
investigative  strategies  were  adopted  to  determine  the  best  teaching  methods.  First,  lessons 
learned  from  the  research  literature  on  training  CT  and  distance-based  learning  were  examined. 
Second,  current  efforts  to  teach  CT  to  Army  officers  were  studied.  Anny  instructors  and 
instructional  developers  involved  in  teaching  CT  were  surveyed  to  identify  their  successes  and 
problems  in  teaching  CT..  The  results  from  the  two  research  strategies  were  then  used  to 
establish  functional  requirements  for  web-based  CT  instructional  material. 

The  functional  requirements  were  used  to  guide  the  design  of  (CT)  .  A  set  of  pedagogical 
principles  were  also  derived  from  the  literature  review,  survey,  functional  requirements,  and 
practical  considerations.  These  served  as  the  philosophical  basis  of  (CT)2’s  design. 

To  evaluate  the  effectiveness,  usability,  and  student  acceptance  of  (CT)“,  one  module  of 
(CT)'  was  assessed.  The  central  objective  of  the  research  was  to  determine  whether  (CT) 
effectively  improves  CT  compared  to  two  other  learning  conditions.  The  research  also  assessed 
participating  students’  attitudes  and  subjective  evaluations  of  (CT)  as  indicators  of  acceptance 
and  usability. 
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Findings: 


Examination  of  the  lessons  learned  from  the  research  literature  on  training  CT  and 
distance-based  learning  yielded  a  number  of  results  that  were  used  to  guide  the  development  of 
(CT)2.  These  are  summarized  here.  The  literature  indicates  that  CT  can  be  successfully  taught  in 
the  classroom  or  as  part  of  a  distance-learning  program.  However,  distance  learning  does  tend  to 
produce  lower  motivational  levels  among  students  who  take  distance  learning  courses  (Eldred, 
1994).  Therefore,  any  attempt  to  teach  CT  using  distance-learning  methods  should  provide 
motivational  components  to  counteract  the  isolation  inherent  in  distance  learning.  For  example, 
multimedia  and  interactivity  can  be  used  to  increase  motivation  and  to  increase  the  efficiency  by 
which  information  is  transmitted  to  the  student.  CT  training  should  also  include  rich  and  varied 
examples,  which  should  be  presented  in  real  world  contexts.  Instruction  should  be  context-rich, 
problem-based,  and  goal-oriented  and  should  include  interactive  simulations  for  demonstration, 
exploration,  and  discovery,  practice  and  for  transfer  of  knowledge  to  new  situations.  The  CT 
tasks  targeted  by  the  training  should  be  selected  from  the  environment  to  which  they  will  be 
applied.  Therefore,  (CT)  training  should  target  those  critical  thinking  skills  that  are  important 
and  problematic  in  Army  environments. 

The  results  of  the  survey  of  Army  instructors  and  curriculum  developers  involved  in 
teaching  and  developing  instruction  for  CT  indicated  that  they  are  mostly  satisfied  with  how  CT 
is  taught  at  the  schoolhouse.  They  believe  that  CT  is  adequately  covered  in  the  curriculum  and 
that  the  guiding  conception  of  CT  that  they  use  (based  on  Paul  and  Elder,  2001)  is  useful  and 
sufficient.  Their  experience  suggests  that  realistic,  practical  exercises  that  require  practicing  CT 
skills  are  the  best  way  to  improve  those  skills.  However,  they  also  say  that  feedback  about 
students’  performance  is  a  necessary  feature  of  practical  exercises.  They  favor  rich  and  varied 
examples  and  realistic  context  for  the  exercises.  Their  central  complaint  as  teachers  and 
curriculum  developers  is  that  the  Army  culture  tends  (1)  not  to  support  the  slow  deliberate 
thought  that  characterizes  CT,  and  (2)  to  discourage  innovative  thinking.  For  this  reason,  some 
instructors  and  curriculum  developers  avoid  talking  about  CT  in  the  abstract.  All  respondents 
agreed  that  it  is  difficult  to  convince  students  of  the  need  and  importance  of  improving  their  CT 
skills. 


The  literature  on  CT  provides  a  different  perspective  of  how  CT  should  be  taught  than  the 
recommendations  from  instructors.  These  two  vantage  points  seem  to  yield  different,  yet 
compatible,  kinds  of  recommendations.  Researchers  and  instructional  experts  agree  that  practical 
exercises  are  the  best  way  to  teach  CT  and  say  that  distance  based  methods  may  be  effective. 
They  agree  that  examples  that  are  realistic,  practical,  and  context-rich  are  effective.  However,  the 
literature  has  more  to  say  about  methods  of  training.  While  instructional  experts  report 
satisfaction  with  current  CT  theories  (Paul  &  Elder,  2001)  and  the  use  of  practical  exercises  as  a 
useful  method,  the  literature  suggests  that  a  number  of  other  techniques  may  be  more  effective. 

The  results  of  the  literature  review  and  instructor  survey  informed  the  establishment  of 
functional  requirements  for  a  distance-learning  program  of  CT  training,  (CT)".  Functional 
requirements  for  such  a  program  were  specified,  including  (1)  an  estimate  of  instructional 
capability,  (2)  scope  of  the  instructional  problem,  (3)  location  of  instruction,  (4)  anticipated 
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student  load,  (5)  alternative  instructional  strategies  methods,  and  media,  (6)  support  for 
providing  and  maintaining  the  system,  (7)  facilities  requirements,  (8)  evaluation  and  assessment, 
(9)  quality  improvement  metrics,  and  (10)  other  requirements  specific  to  teaching  CT  in  an  on¬ 
line  format. 

A  set  of  pedagogical  principles  were  developed  based  on  a  review  of  the  literature  on 
training  critical  thinking  and  on  distance  learning.  These  principles  served  as  the  philosophical 
basis  of  (CT)2’s  design.  The  principles  are: 

♦  CT  skills  can  be  learned,  trained,  and  transferred. 

♦  Practice  is  essential. 

♦  Feedback  is  essential. 

♦  Assessment  is  essential. 

♦  Training  conditions  should  optimize  transfer. 

♦  Part-training  methods  are  most  effective. 

♦  Focus  on  importan  t  and  problematic  critical  thinking  skills. 

♦  Focus  on  common  and  consequential  errors. 

♦  Use  training  methods  appropriate  for  adults  (army  officers). 

♦  Start  with  concrete  experiences.. 

♦  Training  should  be  scenario  based. 

♦  Training  should  be  increasingly  complex  and  difficult. 

♦  Training  should  be  distance-based. 

♦  Multimedia  is  essential. 

♦  Interactivity  is  essential. 

♦  Scoring  increases  motivation  and  should  be  used. 

♦  Use  innovative  training  techniques. 

Based  on  these  principles,  an  on-line  CT  course,  (CT)2,  was  designed  and  developed.  The 
focus  of  (CT)2  is  on  improving  key  critical  thinking  skills  that  are  important  and  problematic  in 
Army  battle  command.  CT  involves  a  deliberate,  systematic  awareness  of  the  process  and 
products  of  one’s  own  thinking.  The  training  program  focuses  on  targeting  common  -  and 
potentially  serious  -  errors  that  people  make  when  they  fail  to  apply  appropriate  critical  thinking 
skills.  These  errors  include  overlooking  important  details,  misinterpreting  information,  and 
making  incorrect  assumptions  -  all  of  which  can  lead  to  poor  situation  assessment,  decision¬ 
making,  and  problem  solving.  The  training  program  highlights  awareness  of  these  errors  and 
teaches  specific  techniques  that  can  help  people  overcome  them.  It  presents  students  with  real- 
world  situations  and  asks  them  to  complete  numerous  thinking  exercises  that  require  the  practice 
and  application  of  CT  skills  in  a  variety  of  realistic  settings. 

The  effectiveness  of  (CT)"  was  assessed  in  an  evaluation  in  which  participants  from  the 
85th  Reserve  Training  Division  worked  through  parts  of  the  training.  Although  the  evaluation 
sample  size  was  small,  (CT)2  was  positively  rated  by  all  participants.  Despite  the  extensive  time 
commitment  the  training  program  requires,  users  found  it  interesting  and  well  worth  their  time. 

(CT)"  appears  to  be  generally  effective  at  encouraging  critical  thinking,  at  least  about 
messages  Army  personnel  must  evaluate.  The  evaluation  showed  that  the  web-based  training 
enhanced  memory  for  messages,  possibly  because  it  encourages  greater  depth  of  processing.  The 
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finding  that  memory  of  messages  is  enhanced  after  participating  in  (CT)"  is  a  strong  indicator 
that  those  messages  have  been  processed  at  a  deeper  level. 

The  results  suggest  that  training  using  (CT)"  inhibits  the  production  of  (potentially 
incorrect)  inferences  that  go  beyond  what  is  explicitly  given  in  the  message.  Participants  who 
took  the  (CT)"  training  made  significantly  fewer  unjustified  inferences  than  participants 
assigned  to  the  other  two  training  conditions.  Examination  of  the  responses  reveals  that  (CT)2 
participants  did  make  inferences;  however,  they  justified  them  by  pointing  out  explicit 
information  given  in  the  message  that  supported  their  inferences.  (CT)"  appears  to  encourage 
discrimination  of  what  is  “known”  or  “given”  from  what  might  be  added  (i.e.,  inferred)  by  the 
perceiver. 

Utilization  and  Dissemination  of  Findings: 

(CT)"  can  be  used  by  the  Army  training  and  education  communities  as  a  tool  to  improve 
critical  thinking  skills  in  Anny  officers.  The  training  program  is  easily  accessed  on  the  web  and 
is  appropriate  for  use  as  stand-alone  training  for  self  development,  use  as  distance  learning  in  a 
school  house  curriculum,  or  in  a  classroom  setting  as  take  home  work  or  for  in-class  exercises. 
Researchers,  teachers,  and  curriculum  developers  can  use  the  findings  as  a  stepping-stone  to  the 
development  of  future  CT  educational  programs,  or  to  better  understand  (CT)2  for  their  own  use 

The  prototype  training  program  described  in  this  volume  provides  training  for  two  CTs.  It 
serves  as  the  pilot  system  for  a  larger  training  program  which  trains  the  full  set  of  eight  CTS. 
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CRITICAL  THINKING  TRAINING  FOR  ARMY  OFFICERS 
VOLUME  THREE:  DEVELOPMENT  AND  ASSESSMENT  OF  A  WEB- 

BASED  TRAINING  PROGRAM 


INTRODUCTION 

Interest  in  promoting  critical  thinking  (CT)  skills  has  increased  over  the  past  20  years  in  a 
variety  of  diverse  applications  such  as  public  education,  military  leadership,  nursing,  technical 
vocations,  and  corporate  business.  Not  surprisingly,  the  United  States  military  is  at  the  forefront 
of  the  effort  to  promote  and  improve  thinking  skills.  Military  leadership  demands  the  application 
of  high  quality  CT  for  effective  battle  command,  where  battle  command  applies  “to  the 
leadership  element  of  combat  power... Commanders  visualize  the  operation,  describe  it  in  terms 
of  intent  and  guidance,  and  direct  the  actions  of  subordinates  within  their  intent.  They  directly 
influence  operations  by  personal  presence...”  (Department  of  the  Anny,  2001.  Section  5-1).  For 
this  reason  a  new  Web-based  CT  training  system,  Computerized  Training  in  Critical  Thinking 
(CT)“,  was  developed. 

The  present  report  is  the  third  in  a  series  of  three  volumes  which  describe  a  research 
program  in  developing  training  for  critical  thinking  which  culminated  in  the  development  of 
(CT)“.  The  first  volume  (Fischer,  Spiker,  &  Riedel,  2008a)  presents  an  overview  of  the  research 
program  that  developed  and  validated  a  theoretical  model  for  the  training,  selected  and  validated 
eight  high  impact  critical  thinking  skills  for  Army  officers,  and  developed  and  evaluated  the 
training  course.  The  second  volume  (Fischer,  Spiker,  &  Riedel,  2008b)  presents,  in  more  detail,  the 
model  and  theoretical  basis  for  the  training  and  the  investigations  that  were  conducted  to  validate 
the  model.  This  volume  describes  the  prototype  training  system  that  was  developed,  including  the 
functional  requirements,  pedagogical  principles,  learning  objectives,  course  organization  and 
content,  and  evaluation  of  training.  A  fourth  report  (Fischer,  Spiker,  Harris,  McPeters,  &  Riedel, 
2008)  describes  the  expanded  training  system  and  its  assessment. 


Promoting  CT  Through  Training 

Advanced  training  in  CT  is  needed  for  adult  populations  in  many  fields  of  work.  Several 
domains  (e.g.,  nursing,  business,  military  leadership)  have  recognized  the  need  to  improve  the 
CT  of  personnel  whose  cognitive  skills  they  suspect  are  deficient  (e.g.,  Hawley,  1998).  It  is  not 
surprising  that  CT  ability  may  be  substandard  because  education  in  nearly  all  areas  has 
traditionally  focused  on  the  accumulation  of  content  knowledge,  often  neglecting  to  teach  the 
reasoning  skills  that  process  such  knowledge.  Tucker  (1996,  p.  46)  remarks  that,  “while  content 
knowledge  is  a  crucial  part  of  this  value  creation,  critical  thinking  skills  are  the  adjudicative 
engines  that  drive  everything  from  boardroom  strategic  decisions  to  the  creative  responses  of  a 
software  company's  help  desk.”  In  short,  education  and  training  may  not  have  kept  up  with 
changes  in  demand. 
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CT  Training  in  the  Military 


Military  leadership  demands  the  application  of  high  quality  CT  for  effective  battle 
command  and  United  States  military  educational  institutions  recognize  the  importance  of 
promoting  and  improving  critical  thinking  skills.  Military  leaders  must  make  tactical  decisions  in 
complex  and  stressful  situations  where  knowledge  is  incomplete  and  uncertain.  The  information 
provided  to  a  battle  commander  is  always  incomplete,  often  inaccurate,  and  sometimes 
purposefully  misleading.  CT  skills  involving  the  evaluation  of  battlefield  infonnation  is  crucial 
under  these  circumstances.  Moreover,  a  leader’s  ultimate  solutions  to  battle  command  problems 
must  be  effective,  yet  not  be  predictable.  Therefore,  he  cannot  simply  base  his  battle  plans  on 
well-learned  battlefield  patterns;  he  must  reason  through  and  integrate  an  enormous  amount  of 
information.  Army  leaders  are  finding  themselves  in  situations  that  bear  little  resemblance  to 
conflict  situations  they  have  previously  experienced  or  studied.  In  such  situations,  novel 
solutions  that  are  the  product  of  CT  will  be  critical  to  success. 

Sometimes  battle  commanders  must  make  rapid  decisions  in  situations  where  there  may 
not  be  time  for  extensive  CT.  Other  times,  however,  a  battle  commander  may  have  several  days 
to  develop  his  plan  and  issue  orders,  which  is  sufficient  time  to  apply  concentrated  thinking.  In 
cases  where  the  ensuing  situation  requires  rapid  decision-making,  even  an  available  20  minutes 
may  provide  an  opportunity  to  apply  CT.  In  fact,  CT  becomes  even  more  important  in  the 
execution,  as  opposed  to  the  planning,  phase  of  battle  because  events  rarely  occur  in  concordance 
with  the  original  plan.  CT  is  not  only  critical  for  commanders  at  all  echelon  levels,  but  also  for 
staff  officers  who  are  responsible  for  summarizing  large  quantities  of  information  from 
numerous  sources,  or  for  making  recommendations  to  the  commander.  In  summary,  battle 
command  is  clearly  a  domain  in  which  CT  is  important  to  performance. 

Extensive  training  time  is  devoted  to  the  accumulation  of  content  knowledge  in  the 
military.  For  example,  every  Anny  officer  studies  Clausewitz’s  principles  of  warfare.  Traditional 
Army  education  offers  a  prescriptive  model  that  corresponds  to  doctrine,  and  focuses  on  the 
products  of  decision-making  (Fallesen,  Michel,  Fussier,  &  Pounds,  1996).  Historically,  relatively 
little  training  time  has  been  spent  on  improving  the  process  of  thinking  and  decision-making1 2.  It 
is  not  that  the  Army  educational  system  has  neglected  to  provide  instruction  in  critical  thinking 
or  reasoning,  but  that  fewer  resources  have  been  devoted  to  the  training  of  thinking  processes 
than  to  other  important  skills.  Moreover,  the  prescriptive  and  procedural  nature  of  the  doctrinal 
methods  may  actually  discourage  the  application  or  development  of  thinking  skills,  inhibiting  the 
creation  of  novel  solutions  that  might  be  the  result  of  CT.  Combat  Training  Center  experiences 
tend  to  reinforce  the  schoolhouse  model  of  doctrinal  decision  making  by  evaluating  unit 
performance  based  on  adherence  to  a  prescriptive  model.  In  so  doing,  they  tend  to  encourage  an 
algorithmic  approach  to  command  decision-making  and  discourage  flexible  and  thoughtful 
approaches  to  meeting  the  objectives  of  the  mission  (Fallesen,  et  ah,  1996). 

If  relatively  few  resources  are  devoted  to  developing  good  thinking  habits,  officers  must 
develop  and  hone  their  own  methods  of  thinking  to  support  decision-making.  Without  explicit 
training,  whatever  thinking  skills  a  military  leader  possesses  are  gained  through  on-the-job 

1  However,  the  new  Intermediate  Level  Education  program  at  the  Army  Command  and  General  Staff  College 
includes  training  in  critical  thinking  that  is  integrated  into  every  lesson  plan. 
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experience,  fortuitous  experiences  in  training  exercises,  individual  disposition,  or  other  means 
such  as  self-study.  Establishing  an  integrated  training  program  to  address  the  development  of 
thinking  skills  in  battlefield  commanders  is  clearly  preferable  to  hoping  that  these  skills  will 
develop  on  their  own.  In  short,  the  education  of  military  commanders  seems  a  ripe  opportunity 
for  developing  training  designed  to  foster  thinking  skills,  and  that  the  domain  of  battle  command 
is  one  in  which  CT  is  crucial.  However,  the  development  of  superior  critical  thinking  skills 
requires  extensive  deliberate  practice  and  sometimes  unpleasant  intellectual  work  (Paul  &  Elder, 
2001).  As  is  necessary  to  develop  any  complex  skill  (e.g.,  a  tennis  serve),  developing  proficient 
CT  habits  also  requires  excellent  coaching,  internal  motivation,  self-awareness,  and  the  ability  to 
critically  evaluate  one’s  own  perfonnance  (Paul  &  Elder,  2001). 

Development  of  training  that  effectively  and  efficiently  improves  critical  thinking  skills 
that  underlie  proficient  battle  command  rests  on  the  ability  to  identify  such  skills.  Before 
effective  training  can  be  developed,  training  objectives  must  be  specified,  and  those  objectives 
should  be  derived  from  an  empirically  tested  model  of  CT.  In  the  second  volume  of  this  series 
(Fischer,  et  al.,  2007b),  such  a  model  was  described. 

This  CT  model  has  been  used  to  guide  two  major  efforts  to  develop  CT  training 
curriculum.  The  Intermediate  Educational  Level  (ILE)  team  at  the  Command  and  General  Staff 
College  (CGSC)  at  Fort  Leavenworth,  Kansas,  conducted  the  first  effort.  In  the  summer  of  2000, 
the  ILE  training  development  team  was  tasked  with  developing  new  curriculum.  The  ILE  team 
based  their  CT  training  on  the  model  of  CT  and  the  associated  research  described  in  Volume 
Two  of  this  series,  which  identifies  a  set  of  eight  critical  thinking  skills  that  are  important  and 
problematic  to  Army  battle  command. 

The  CT  model  was  also  used  to  guide  development  of  a  new  training  program  that 
promotes  CT  in  Army  officers.  Research  results  reported  in  Volume  Two  (Fischer,  et  al.,  2008b) 
indicated  that  additional  training  was  needed  that  could  be  easily  and  widely  accessed  by  Army 
officers  throughout  the  world.  A  needs  analysis  indicated  that  the  training  should  be  distance- 
based  and  a  new  Web-based  CT  training  system  was  developed,  Computerized  Training  in 
Critical  Thinking  (CT)2.  The  present  volume  documents  the  research  and  development  effort  that 
produced  the  prototype  version  of  (CT)'.  This  volume  also  documents  the  pedagogical  strategy, 
learning  objectives,  organization,  and  lessons  of  the  web-based  training.  Finally,  it  describes  and 
discusses  the  results  of  an  evaluation  of  (CT)'. 


3 


LESSONS  LEARNED  ABOUT  TRAINING  CRITICAL  THINKING 


The  primary  goal  for  (CT)2  was  easily  accessed,  state-of-the-art  distance-learning 
instruction  that  would  effectively  train  critical  thinking  skills  (CTS)  for  Army  officers..  The  first 
step  in  meeting  this  goal  was  the  identification  of  CTS  to  train.  To  address  this  issue,  research 
(Fischer,  et  ah,  2008b)  identified  a  set  of  critical  thinking  skills  as  important  and  problematic  to 
Army  officers.  In  developing  (CT)"  ,  it  was  also  important  to  employ  the  best-known  techniques 
for  promoting  CT  and  distance-based  training.  The  second  step  then  in  developing  (CT)2  was  the 
identification  of  teaching  methods  that  are  likely  to  be  most  effective  at  developing  CT  in  Army 
officers. 

Two  investigative  strategies  were  adopted  to  determine  the  best  methods.  First,  lessons 
learned  from  the  research  literature  on  training  CT  and  distance-based  learning  were  studied. 
Identification  of  lessons  learned  from  others’  experiences  was  an  efficient  and  cost-effective 
method  to  avoid  blind  alleys,  recapture  earlier  successes,  and  gain  low-cost  insights  into 
potential  problem  areas.  The  education  and  psychology  literature  was  reviewed  to  detennine  the 
types  of  CT  skills  for  which  training  has  been  developed,  the  methods  used  to  teach  such  skills, 
and  the  methods  that  have  been  shown  to  be  most  effective. 

Second,  current  efforts  to  teach  CT  to  Army  officers  were  studied.  Anny  instructors  and 
instructional  developers  who  are  involved  in  teaching  CT  were  surveyed  to  identify  the  successes 
and  the  problems  they  experienced  related  to  training  CT.  The  survey  focused  on  the  content  and 
organization  of  current  CT  courses  as  well  the  instructors’  experiences  and  opinions  about 
teaching  CT  to  Army  officers.  The  survey  was  designed  to  reveal  lessons  learned  from  current 
efforts  to  promote  CT  within  the  military  training  system. 

The  results  from  the  two  research  strategies  were  then  used  to  establish  functional 
requirements  for  web-based  CT  instructional  material,  which  are  discussed  later  in  this  report. 
The  methods,  results,  and  conclusions  of  the  CT  training  literature  review  and  examination  of 
current  teaching  methods  are  provided  below. 

Lessons  Learned  from  Past  Research 

What  methods  have  been  used  and  evaluated  for  teaching  CT?  What  kinds  of  CT  skills 
have  educators  and  researchers  targeted?  Which  methods  are  most  effective  for  promoting  CT? 
How  can  the  best  teaching  methods  be  implemented  in  a  distance  learning  application?  To 
answer  these  questions,  a  literature  review  was  conducted  that  encompassed  academic  sources 
from  the  fields  of  education  and  psychology. 

A  variety  of  key  words  were  used  to  search  key  databases,  each  relevant  to  development  of 
web-based  training  of  CT.  Key  words  were  combined  to  narrow  the  search  to  articles  relevant  to 
training  and  education,  with  a  particular  emphasis  on  distance,  computer-based,  and  web-based 
instruction,  such  as  “critical  thinking,”  “training,”  “education,”  “instruction,”  web-based,” 
“computer-based,”  and  “distance  learning.”  The  Internet  was  also  searched  for  sites  pertaining  to 
web-based  training,  distance  learning,  and  CT  training.  This  search  strategy  produced  over  1000 
hits.  When  redundancies  among  item  content  were  eliminated,  the  list  of  sources  to  review  was 
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reduced  to  134  sources.  However,  many  of  the  sources  were  only  tangentially  related  to  training 
CT.  Of  the  134  sources,  43  (books,  articles,  reports,  and  websites  pages)  directly  related  to  the 
issue  of  teaching  CT  were  obtained  and  reviewed.  An  additional  29  sources  relating  to  distance 
learning,  web-based  training,  and  computer-based  training  were  reviewed.  Below,  we  summarize 
the  findings  and  conclusions  of  the  review,  organized  by  the  central  questions  we  sought  to 
resolve. 

What  Critical  Thinking  Skills  Are  Taught? 

Volume  Two  of  this  series  (Fischer,  et  ah,  2008b)  discusses  the  critical  thinking  skills  that 
theorists  have  proposed  are  important  to  common  cognitive  tasks  and  contains  a  list  of  critical 
thinking  skills  appearing  in  the  literature.  For  convenience  sake,  this  list  is  reprinted  in  Appendix 
A  of  this  volume.  The  skills  listed  in  Appendix  A  were  used  to  organize  the  information  that  we 
gathered  in  the  literature  review  about  CT  training  methods.  The  skill  that  was  discussed  and/or 
trained  in  each  source  was  first  identified  as  one  of  the  CT  skills  listed  in  Appendix  A.  The 
teaching  method(s)  used  to  promote  each  critical  thinking  skill  was  then  recorded. 

Appendix  B  of  this  report  summarizes  the  kinds  of  skills  for  which  researchers  have 
developed  training  materials  or  techniques  and  the  teaching  methods  used  for  each  skill. 
Appendix  B  also  identifies  the  reference  from  which  each  training  method  was  identified,  and 
the  kind  of  evidence  provided  by  each  source.  It  is  clear  from  Appendix  B  that  educational 
researchers  have  attempted  to  teach  a  wide  variety  of  critical  thinking  skills  (CTS).  However,  it 
is  also  clear  that  training  methods  have  not  been  developed  for  many  of  the  skills  listed  in 
Appendix  A.  For  example,  argument  analysis  skills  have  frequently  been  targeted  by  teaching 
students  to  distinguish  among  assumptions,  evidence,  and  conclusions,  and  by  having  students 
critique  arguments.  Meta-critical  thinking  skills,  however,  have  received  much  less  attention. 
Therefore,  of  the  many  possible  critical  thinking  skills  that  could  be  developed  through  training, 
only  a  few  have  been  systematically  targeted.  It  appears  that  scientists  and  educators  have 
focused  on  developing  only  a  small  set  of  CTS. 

It  is  not  clear  why  some  CTS  have  received  attention  and  others  have  not.  Researchers 
have  a  variety  of  purposes,  biases,  and  preferences,  which  may  drive  their  interest  in  some  CTS 
but  not  others.  Little  attempt  has  been  placed  on  establishing  the  relative  importance  of  CTS, 
which  may  vary  depending  on  the  environment  in  which  they  are  applied.  Little  attempt  has  been 
made  to  organize  the  CTS  in  some  meaningful  way  that  would  facilitate  their  selection  for 
training. 

What  Methods  Are  Used  To  Teach  CT? 

To  simplify  the  diverse  teaching  methods  identified  in  Appendix  B,  each  was  classified  as 
one  of  seven  general  methods,  which  include:  (1)  overt  argument  analysis,  (2)  applied  argument 
analysis,  (3)  evidence-based  or  scientific  analysis,  (4)  meta-cognitive  strategies,  (5)  question 
asking  or  the  Socratic  method,  (6)  group  discussion,  and  (7)  guided  practice.  These  seven  general 
categories  are  defined  below. 
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Argument  analysis  is  a  popular  and  common  teaching  strategy  used  to  promote  CT. 
Argument  analysis  can  be  either  overt  or  applied.  Overt  argument  analysis  (e.g.,  Bernstein, 

1995)  requires  students  to  evaluate  competing  arguments.  It  explicitly  teaches  students  about 
concepts  such  as  assumptions,  claims,  evidence,  etc.  For  example,  students  may  be  taught  to 
analyze  and  evaluate  competing  arguments  on  an  issue.  Instructors  may  then  have  their  class 
engage  in  discussions  about  the  arguments.  Another  example  of  overt  argument  analysis  is 
explicit  instruction  in  standards  for  forming  generalizations.  In  contrast,  methods  that  use  applied 
argument  analysis  (Colbert,  1986)  are  embedded  in  activities  such  as  negotiation,  debate,  and 
generating  predicted  consequences  of  actions  within  specified  topic  areas.  For  example,  students 
may  learn  to  analyze  arguments  by  engaging  in  a  negotiation  or  a  debate  process  with  another 
student  who  has  taken  an  opposite  position  on  a  particular  issue. 

Methods  that  use  evidence-based  analysis  (Zobar,  Weinberger,  &  Tamir,  1994) 
emphasize  the  logic  inherent  in  the  scientific  method.  In  such  methods,  analysis  of  experiments, 
problem  solving,  and  data  handling  may  be  taught.  For  example,  some  researchers  suggest  that 
the  explicit  teaching  of  experimental  research  methods  increases  CT.  Another  kind  of  method, 
Meta-cognitive  strategy,  teaches  students  how  to  think  about  their  own  thinking.  For  example, 
Klein,  Olson,  and  Stanovich  (1997)  taught  students  to  plan  for  monitoring  their  evaluation  of  an 
argument  by  reminding  themselves  that  they  would  either  be  a  “careful  idea  shopper,”  or  “try  to 
figure  out  what  the  author  wants  me  to  believe.”  Paul  and  Elder’s  widely  used  CT  textbook 
(2001)  also  encourages  meta-cognitive  development  by  having  students  think  about  their  current 
habits  of  thinking.  Each  chapter  contains  several  exercises  in  which  students  are  directed  to 
examine  some  component  of  their  habitual  thought  processes. 

Another  popular  method  for  promoting  CT,  the  Socratic  method  (Jones  &  Safrit,  1994), 
requires  students  to  answer  or  generate  questions.  The  Socratic  teaching  method  is  a  loosely 
related  family  of  pedagogical  techniques  that  vary  substantially  in  purpose,  design,  and  outcome. 
The  only  thread  that  unites  these  methods  is  the  inclusion  of  questions.  In  its  original  form,  the 
Socratic  method  was  used  to  illuminate  the  logical  fallacies  of  an  individual’s  beliefs  and 
attitudes.  It  was  a  method  that  could  only  be  applied  “one-on-one,”  because  it  was  designed  to 
make  a  single  student  question  his/her  positions.  Socrates  himself  claimed  to  be  ignorant  of  the 
“truth”  and  would  ask  the  student  questions  to  detennine  the  veracity  of  positions  the  student 
held.  Socrates  then  examined  the  logical  consequences  of  each  answer.  The  usual  conclusion  of  a 
Socratic  session  was  to  show  the  contradictory  assumptions  and  conclusions  inherent  in  the 
reasoning  held  by  the  student.  In  so  doing,  Socrates  improved  the  CT  of  a  particular  student  by 
encouraging  reexamination  of  his/her  argument. 

Teachers  have  modified  Socrates’  original  method  in  many  different  ways.  For  example, 
questioning  techniques  may  be  used  to  motivate  students  rather  than  to  illuminate  logical 
inconsistencies  in  their  students’  beliefs  (Hittner,  1999).  Law  schools  are  famous  for  the  use  of 
questioning  to  promote  motivation,  where  students  must  be  well  prepared  to  answer  any  question 
in  the  classroom,  or  risk  humiliation.  Law  schools  also  use  questioning  to  encourage  students  to 
back  up  their  reasoning  with  evidence.  “Socratic”  methods  also  vary  in  the  types  of  questions 
that  are  posed  to  students.  Some  techniques  use  questions  that  have  factual  answers  that  students 
can  easily  provide.  Others  use  questions  that  require  creative  answers  because  there  is  no  single 
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correct  response.  While  Socrates  applied  his  technique  to  individuals,  current  questioning 
methods  are  typically  applied  in  a  group  discussion  format. 

In  fact,  Group  discussion  (e.g.,  Garside,  1996)  is  the  most  common  means  used  to 
promote  CT  in  the  classroom.  Group  discussion  is  thought  to  improve  CTS  by  exposing  students 
to  perspectives  other  than  their  own.  Other  social  teaching  methods  used  to  promote  CT  include 
collaborative  learning,  the  introduction  of  controversy  in  class  discussion,  and  group  assignments 
that  ask  students  to  identify  causes  of  events,  develop  explanations  for  phenomena,  etc. 

Guided  practice  (e.g.,  Jones  &  Safrit,  1994),  another  method  used  to  improve  CTS, 
requires  students  to  practice  CT  skills  under  the  tutorship  and  guidance  of  a  teacher  who  also 
models  performance.  For  example,  students  may  be  asked  to  role  play  or  dramatize  a  decision 
maker  or  problem  solver. 

Certain  kinds  of  teaching  methods  are  probably  better  suited  for  improving  some  critical 
thinking  skills  than  for  others.  For  example,  the  Socratic  method  may  be  more  appropriate  for 
tasks  that  involve  the  creative  generation  of  solutions  such  as  inductive  reasoning.  Alternatively, 
overt  argument  analysis  teaching  methods  are  probably  more  effective  at  promoting  tasks  that 
involve  deductive  reasoning.  Unfortunately,  little  is  known  about  which  methods  are  most 
effective  at  teaching  particular  tasks.  Below,  we  discuss  the  current  state  of  knowledge  about  the 
effectiveness  of  these  teaching  methods. 

Which  Teaching  Methods  Are  Most  Effective? 

Appendix  B  shows  that  a  small  proportion  of  the  methods  that  have  been  used  to  promote 
CT  have  been  empirically  validated.  Most  of  the  sources  we  reviewed  described  pedagogical 
methods  that  were  employed  in  existing  classrooms.  Many  failed  to  report  quantitative,  or  even 
qualitative,  measures  of  perfonnance  after  the  training  was  delivered.  The  literature  cited  in 
Appendix  B  contains  a  large  number  of  opinion  papers  and  testimonials  assuring  readers  that,  in 
the  author’s  experience,  the  methods  work  well  to  increase  CT.  Unfortunately,  testimonial 
evidence  is  often  biased.  If  empirical  evidence  is  applied  as  a  standard,  we  must  conclude  that 
current  research  lacks  information  about  the  effectiveness  of  CT  teaching  methods. 

A  second  problem  is  that  many  of  the  empirical  investigations  summarized  in  Appendix  B 
have  been  conducted  on  training  effectiveness  are  large-scale  program  evaluations  (e.g., 
Herrnstein,  Nickerson,  de  Sanchez,  &  Swets,  1986;  Klein,  Olson,  &  Stanovich,  1997).  For 
example,  Herrnstein,  et  ah,  (1986)  implemented  a  large-scale  educational  program  in  Venezuela 
that  included  6  schools  and  over  400  students.  The  educational  program  was  designed  to  increase 
Venezuelan  school  children’s  levels  of  thinking,  and  used  many  methods  to  achieve  its  goals.  In 
one  method,  students  were  given  explicit  and  overt  training  in  reasoning  and  guided  practice.  In 
another,  CT  training  was  embedded  in  content  courseware  and  instructional  methods.  Although 
findings  of  the  research  were  robust  and  found  that  clear  improvements  had  been  made  in  CT, 
the  evaluation  design  precluded  determining  which  methods  were  effective.  Programmatic 
evaluations  such  as  the  Herrnstein,  et  ah,  (1986)  research  do  little  to  help  instructional  designers 
who  are  seeking  ways  to  include  CT  training  in  their  curriculum  because  they  fail  to  identify  the 
specific  techniques  are  effective  in  training  CT. 
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A  third  problem  is  that  the  CT  training  literature  is  as  fragmented  as  the  CT  theoretical 
literature,  primarily  due  to  disagreement  on  the  definition  of  CT.  Researchers  are  eclectic  when 
choosing  CT  skills  on  which  to  focus  their  training.  Some  focus  on  a  single  skill,  such  as  making 
reasoned  valued  judgments,  because  they  limit  CT  to  this  narrow  definition.  Others  train  a  host 
of  skills  that  typically  fall  into  the  category  of  reasoning  because  they  see  CT  as  a  cognitive 
activity  that  primarily  involves  formal  and  infonnal  logic.  The  literature  is  also  fragmented 
because  the  teaching  methods  and  techniques  reported  are  largely  idiosyncratic  to  the  particular 
author  reporting  the  technique.  Thus,  very  little  overlap  appears  among  teaching  methods.  The 
fragmentation  of  the  training  literature  makes  it  difficult  to  compare  studies. 

In  conclusion,  in  our  opinion  based  on  the  review  summarized  in  Appendix  B,  current 
research  lacks  empirical  evidence  that  would  guide  selection  of  teaching  methods  to  promote  CT. 
However,  the  literature  does  provide  the  instructional  developer  with  a  set  of  potential  methods 
from  which  to  select  a  pedagogical  strategy.  New  efforts  to  train  CT  could  consider  argument 
analysis,  evidence-based  analysis,  meta-cognitive  strategies,  questioning,  group  exercises,  and 
modeling  as  approaches  that  might  improve  CT. 

Many  of  these  methods  are  typically  applied  to  group  settings  and  capitalize  on  the 
individual  differences  inherent  in  groups.  Methods  that  could  be  taught  individually  (e.g., 
argument  and  evidence-based  analysis)  are,  nonetheless,  most  often  used  in  social  situations.  The 
relative  contribution  of  natural  group  dynamics  to  the  promotion  of  CT  is  not  known.  While 
some  researchers  (e.g.,  Bernstein,  1995)  claim  that  group  discussion  is  most  effective  at 
improving  CT,  we  do  not  know  if  the  true  source  of  improvement  is  the  group  dynamics 
inherent  in  the  situation  or  the  teaching  strategies  used. 

Distance  Learning.  The  question  of  whether  CT  can  be  improved  using  distance  learning 
techniques  is  unresolved  at  this  point.  It  is  important  to  consider  the  effectiveness  of  distance 
learning,  notably  computer-based  training,  as  it  has  been  applied  to  the  development  of 
knowledge  and  skills  of  other  domains.  A  brief  review  of  the  literature  on  distance  learning  is 
provided  here. 

Distance  learning  has  traditionally  referred  to  situations  in  which  the  teacher  and  learner 
are  separated  by  time  and/or  space,  with  communication  occurring  through  the  mail,  by  phone, 
or  by  video.  More  recently,  the  term  has  been  broadened  to  refer  to  any  arrangement  in  which 
instruction  occurs  through  a  medium  other  than  face-to-face  (Imel,  1998).  The  learner  may  reside 
in  the  same  town  and  may  even  be  a  student  on  campus  and  still  be  a  distance  learner  if  he  or  she 
accesses  course  materials  on  the  computer  or  interacts  with  the  teacher  through  the  Internet. 

While  Edelson  (1998)  claims  that  Internet  courses  have  "clearly  emerged  as  the 
technology-of-choice"  among  part-time  adult  students  who  are  unable  to  attend  conventional 
classes,  Ostendorf  (as  cited  in  Imel,  1998)  declares  the  most  popular  and  fastest  growing  distance 
education  delivery  mode  in  the  US  is  live  video  instruction.  Other  forms  of  distance  education 
include  synchronous,  interactive  audio  conferencing  over  phone  lines,  and  asynchronous,  non¬ 
interactive  print  or  video  recordings  (Mylona,  1999). 
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Most  studies  comparing  the  effectiveness  of  distance  learning  with  traditional  instruction 
have  found  no  difference  in  learning  outcomes  (Barry  &  Runyan,  1995;  Hiltz,  1990;  Schlosser  & 
Anderson,  1994).  Clark  (1994)  concludes  that  it  is  not  whether  education  is  provided  at  a 
distance  that  makes  the  difference  in  student  learning  but  the  quality  of  the  instruction  itself.  One 
disadvantage  of  distance  learning  is  that  students  commonly  feel  disconnected  from  others  taking 
the  course  and  from  the  instructor,  which  can  cause  anxiety.  Perhaps  this  is  one  factor  that  causes 
the  much  higher  dropout  rate  associated  with  distance  learning.  However,  student  support  groups 
can  reduce  feelings  of  isolation. 

Distance  learning  also  requires  more  self-motivation  from  the  learners  than  traditional 
educational  methods  (Eldred,  1994).  Students  must  maintain  their  attention  and  concentration  on 
the  material  to  be  learned  without  the  motivating  presence  of  an  instructor.  However,  well- 
designed  computer-based  instruction  could  provide  motivation  in  other  ways,  as  could  student 
support  groups  or  learning  teams. 

One  type  of  distance  learning,  computer-based  instruction  (CBI)  or  computer-based 
training  (CBT),  has  been  shown  to  significantly  improve  student  achievement.  A  1990  US 
Institute  for  Defense  analysis  of  47  studies  comparing  the  use  of  interactive  computer  videodisc 
education  with  traditional  classroom  education  found  CBT  more  effective  for  knowledge  and 
performance,  with  lower  attrition  rates  and  instructional  costs  and  a  3 1%  reduction  in  learning 
time  (Fletcher,  1990).  Student  learning  data  from  the  University  of  Melbourne  suggest  enhanced 
learning  through  CBI  (Wills  &  McNaught,  1996).  Other  studies  have  shown  CBT  to  be  at  least 
as  effective  as  traditional  learning,  with  large  decreases  in  learning  time  observed  (Riding  & 
Chambers,  1992). 

Effective  CBT  is  not  just  text  on  the  screen.  The  use  of  interactivity  and  multimedia  are 
crucial  for  high  learning  outcomes.  Interactivity  includes  the  traditional  instructional  features  of 
stimulus,  response,  and  feedback,  as  well  as  computer-based  simulations  in  which  students  can 
manipulate  systems  and  observe  the  results  of  their  actions.  Interactive  simulations  provide 
opportunities  for  active  and  constructive  learning,  and  help  make  abstract  concepts  concrete 
(Rieber  &  Parmley,  1995).  Micro  worlds  are  elaborate  simulated  environments  in  which  students 
explore  to  learn  about  specific  domains.  The  use  of  ThinkerTools,  a  micro  world  designed  to 
teach  Newtonian  physics,  resulted  in  sixth-grade  students  who  outperfonned  high  school 
students  in  a  traditional  physics  class  (White,  1993).  Other  examples  include  a  simulated 
cardiovascular  system  that  reduced  misconceptions  in  biology  students  (Windschitl  &  Andre, 
1998)  and  an  interactive  simulation  of  projectile  motion,  which  resulted  in  significant 
improvement  in  physics  knowledge  (Rieber  &  Pannley,  1995). 

Multimedia  is  the  combination  of  different  presentation  modes,  such  as  words  and  pictures, 
and  different  sensory  modalities,  such  as  acoustic  and  visual.  Mayer  (1997)  reviewed  an 
extensive  series  of  studies  that  support  the  superiority  of  learning  with  multimedia,  including  the 
coordinated  presentation  of  explanatory  words  and  pictures  and  the  simultaneous  presentation  of 
narration  and  animation.  In  an  examination  of  six  controlled  environments  directly  comparing 
traditional  classroom  instruction  to  equivalent  interactive  multimedia  instruction  at  companies 
such  as  IBM  and  Xerox,  CBI  resulted  in  56%  more  learning  in  38-70%  less  time. 


9 


An  advanced  form  of  CBT  is  the  intelligent  tutoring  system  (ITS).  ITS  is  different  from 
standard  CBT  in  that  it  incorporates  an  ongoing  analysis  of  the  student's  ability  and  specific 
instructional  needs,  continually  modifying  the  instruction  to  fit  those  needs.  There  is  currently 
controversy  over  the  appropriateness  of  a  student  model,  with  constructivists  claiming  that 
knowledge  is  not  fixed  but  is  constantly  changing  within  the  context  of  new  experiences  and 
ideas.  However,  the  usefulness  of  a  student  model  for  enabling  systems  to  adapt  to  learners' 
needs  has  been  strongly  supported  (Akhras  &  Self,  2000).  Shute  and  Psotka  (1994)  present 
studies  of  six  existing  ITSs,  each  of  which  have  demonstrated  accelerated  learning. 

With  the  current  attitude  in  education  leaning  toward  the  desirability  of  collaborative 
learning  environments,  in  which  students  interact  and  share  knowledge  and  view  points,  effective 
CBT  will  need  to  find  a  way  to  incorporate  student  collaboration.  However,  the  bottom  line  with 
respect  to  the  controversy  over  CBT  vs.  traditional  classroom  instruction  is  the  consensus  that 
good  quality  instructional  design  will  result  in  high  levels  of  learning  regardless  of  the  delivery 
format. 

CBT  and  distance  learning  are  combined  in  Web-based  training  (WBT).  WBT  incorporates 
the  computer  and  the  Internet  to  provide  instruction  and  communication  to  distance  learners. 
Typically,  CBT  is  downloaded  from  a  website  to  the  learner’s  computer  or  run  directly  from  the 
server.  However,  in  some  instances  training  can  be  directly  accessed  from  web  pages. 

Advantages  of  WBT  include  the  ease  of  revising  curricula,  consistency  in  instruction  across 
distributed  locations  (e.g.,  military  bases),  and  the  convenience  of  the  learner's  having  access  to 
the  instruction  at  any  time.  A  further  advantage  is  the  ability  to  provide  opportunities  for 
collaboration  among  students  via  chat  rooms,  bulletin  boards,  list  servers,  and  e-mail,  if 
appropriate  for  the  training  (Palloff  &  Pratt,  1999). 

A  key  issue  for  CBT  and  WBT  is  the  effect  of  computer  skills  and  typing  skills.  Those 
who  are  not  as  comfortable  using  computer  interfaces  or  a  mouse  will  be  at  a  disadvantage  in 
navigating  the  environment  and  manipulating  on-screen  objects.  Those  with  lesser  typing  skills 
will  be  at  a  disadvantage  in  on-line  communications,  particularly  in  real-time  chat  or  instant 
messaging  situations.  However,  with  the  rapid  spread  of  computers  in  schools,  businesses,  and 
homes,  lack  of  computer  and  typing  skills  will  be  a  decreasingly  important  problem. 

Another  issue  for  WBT  is  that,  in  our  opinion,  the  Internet  currently  provides  much  less 
interactive  and  multimedia  capability  than  CBT.  Internet  browsers  are  partly  to  blame  for  this 
limitation.  They  are  designed  to  deliver  information  in  a  page-by-page  format,  which  may 
constrain  the  design  of  some  training  packages.  They  also  do  not  provide  the  functionality 
offered  by  other  programming  languages  used  in  CBT.  Limits  in  bandwidth  on  the  Internet  are 
also  to  blame.  For  example,  the  slowness  of  dial-up  access  to  the  Internet  would  severely  limit 
the  number  of  large  sound  files  and  graphics  that  one  might  want  to  include  in  WBT.  However, 
technology  advances  may  soon  eliminate  these  limitations. 

Conclusions 

In  conclusion,  there  is  no  reason  to  believe  that  CT  can  be  taught  only  in  the  classroom. 
Distance  learning  has  been  effectively  used  to  promote  better  learning  of  complex  cognitive 
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skills  compared  to  traditional  classroom  methods.  Research  has  shown  that  distance  learning  can 
be  superior  to  other  forms  of  training.  Therefore,  it  is  likely  that  it  can  also  be  effectively  used  to 
improve  CT  in  adult  and  juvenile  populations.  However,  distance  learning  does  produce  lower 
motivational  levels  among  students  who  partake  in  it.  Thus,  any  attempt  to  teach  CT  using 
distance-learning  methods  should  provide  motivational  components  to  counteract  the  isolation 
inherent  in  distance  learning.  Multimedia  and  interactivity  should  also  be  used  to  increase 
motivation  and  to  increase  the  efficiency  by  which  infonnation  is  transmitted  to  the  student.  If  an 
adequate  level  of  interactivity  can  be  provided,  as  detennined  by  the  instructional  goals  of  the 
training,  WBT  can  be  used  to  effectively  teach  CT.  WBT  facilitates  the  use  of  chat  rooms,  e- 
mail,  bulletin  boards,  which  can  provide  the  student  with  a  social  network  needed  to  maintain 
interest  and  combat  isolation.  WBT  is  also  easy  to  modify  and  distribute  in  a  timely  fashion. 

Recent  instructional  theories  indicate  that  CT  training  should  include  rich  and  varied 
examples,  which  should  be  presented  in  real  world  contexts.  All  of  these  desirable  educational 
elements  discussed  above  can  be  incorporated  into  a  WBT  program.  The  training  program  should 
be  a  highly  interactive  and  multimedia,  computer-based  system.  Instruction  should  be  context- 
rich,  problem-based,  and  goal-oriented,  and  should  include  interactive  simulations  for 
demonstration,  exploration,  and  discovery,  and  some  collaborative  environments  for  practice  and 
transfer  of  knowledge  to  new  situations. 

The  critical  thinking  skills  targeted  by  the  training  should  be  selected  for  the  environment 
in  which  they  will  be  applied.  For  Army  officers,  training  should  target  those  skills  important 
and  problematic  to  the  Army.  Moreover,  training  methods  should  be  selected  that  will  be  most 
effective  for  the  particular  task  to  be  trained.  For  example,  if  one  were  to  train  the  task,  Develop 
a  plausible  explanation,  one  should  focus  on  evidence-based  training  techniques  that  will  help 
the  student  evaluate  evidence  and  alternative  explanations  of  that  evidence.  In  contrast,  guided 
practice  might  be  used  to  identify  weak  or  ambiguous  components  of  a  message  to  train  the  skill 
Frame  the  Message. 

Lessons  Learned  from  Army  Instructors  and  Instructional  Developers 

The  Army  currently  incorporates  training  of  CT  in  its  schoolhouse  curriculum  (Bralley, 
2006).  Although  not  as  systematic  or  universal  as  the  schoolhouse  training,  CT  education  also 
occurs  informally  in  field  units  through  experiences  and  mentoring.  Examination  of  current 
methods  of  teaching  CT  is  important  so  as  to  capitalize  on  lessons  learned.  The  research 
described  below  was  conducted  to  assess  instructors’  experience  in  the  teaching  of  CT. 

Method 

Participants.  Eight  instructors  and  instructional  developers  participated  in  the  survey. 
Respondents  were  Anny  officers  or  civilian  researchers  currently  working  at  CGSC  at  Fort 
Leavenworth,  Kansas  State  University,  or  the  Anny  War  College,  The  CGSC  instructors  taught 
at  the  School  for  Command  Preparation  (SCP),  the  Command  General  Staff  Officers  Course 
(CGSOC),  and  the  Combined  Arms  Services  Staff  School  (CAS  ).  The  respondent  from  Kansas 
State  University  was  a  Professor  of  Military  Science. 
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Ten  potential  respondents  to  the  survey  were  originally  identified  from  their  participation 
in  a  Critical  Thinking  Workshop  sponsored  by  the  Army  Research  Institute  at  Fort  Leavenworth, 
Kansas  (Riedel,  Morath,  &  McGonigle,  2001).  .  Each  of  the  ten  were  contacted  and  recruited 
through  phone  and  e-mail  messages,  and  eight  agreed  to  volunteer  their  time  to  participate  in  the 
investigation. 

Three  respondents  were  Lieutenant  Colonels,  and  one  was  a  Major.  Most  of  the  civilian 
respondents  were  researchers  and  instructional  developers  at  CGSC  and  one  was  an  instructor  at 
the  Army  War  College.  Their  years  of  experience  teaching  CT  to  Army  officers  ranged  from  1-4 
years.  Each  held  either  a  masters  or  a  doctoral  degree.  Instructors  typically  taught  only  one 
course,  but  most  had  taught  at  least  six  courses  that  involved  CT.  Instructional  developers  had 
been  involved  in  the  development  of  several  CT  curricula. 

Survey  Materials.  Respondents  completed  a  three-section  survey  that  assessed 
demographic  infonnation  and  their  work  experience  teaching  or  developing  curricula  for  CT,  the 
classes  they  taught  that  involved  CT,  and  their  opinions  and  experiences  in  teaching  CT.  The 
first  section  of  the  survey  asked  respondents  to  provide  infonnation  about  their  educational 
background,  the  number  of  years  they  had  taught  CT,  and  their  current  duty  position.  The  second 
section  contained  eleven  items  that  addressed  respondents’  current  approaches  to  teaching  CT. 
The  open-ended  items  solicited  information  about  the  guiding  theory  used  to  develop  the 
curricula  in  their  courses;  whether  CT  was  taught  overtly,  covertly,  in  the  abstract,  or  in  the 
context  of  military  situations;  the  CT  skills  that  are  taught;  the  format  of  delivery  of  the  course 
materials;  the  duration  and  organization  of  the  course;  and  assessment  measures  and  feedback 
used  in  the  course.  The  third  section  requested  lessons  learned  from  the  instructors’  and 
developers’  experience  in  teaching  CT.  The  items  solicited  opinions  and  judgments.  For 
example,  instructors  were  asked  if  CT  was  adequately  covered;  whether  the  skills  transferred  to 
the  field;  whether  the  guiding  theory  was  adequate;  whether  CT  should  be  taught  overtly, 
covertly,  in  the  abstract,  or  in  the  context  of  military  situations;  and  whether  assessment  tools 
and  feedback  are  adequate.  They  were  also  asked  to  identify  their  successes  and  difficulties  in 
teaching  CT  to  Army  personnel. 

Procedure.  Respondents  were  sent  and  asked  to  complete  the  survey  material.  When  they 
had  completed  the  survey,  they  returned  their  written  responses  and  contacted  the  researchers. 

An  interview  was  then  scheduled.  During  the  interview,  the  survey  administrator  asked  each 
respondent  to  clarify  and  elaborate  those  answers  that  were  not  completely  understandable  from 
the  written  responses. 

Results 

The  survey  results  are  given  below,  organized  by  the  survey’s  two  major  questions.  First, 
we  discuss  participants’  responses  concerning  CT  courses  they  currently  teach.  Second,  we 
report  the  lessons  instructors  and  instructional  developers  learned  from  their  experiences. 
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Lessons  Learned  from  Teaching  CT  in  the  Army.  The  respondents’  answers  to  the  third 
section  of  the  survey  were  highly  informative  about  the  nature  of  teaching  CT  to  Army  officers. 
The  first  question  in  the  survey  addressed  the  adequacy  of  the  CT  instruction  in  the  Anny. 

Is  CT  adequately  covered  in  the  present  curriculum?  Each  instructor  felt  that  within  the 
courses  they  taught,  CT  was  adequately  covered.  However,  each  also  believed  that  CT 
instruction  needed  to  be  extended  to  and  embedded  within  the  Army  curriculum  as  a  whole. 
Particular  need  was  seen  for  instruction  at  junior  officer  levels. 

Does  CT  transfer  to  the  combat  environment?.  Respondents  provided  equivocal  answers  to 
this  question.  Generally,  it  is  impossible  to  know  whether  the  skills  that  are  taught  in  the 
schoolhouse  transfer  to  the  combat  environment  because  assessment  measures  have  not  been 
developed  and  applied.  However,  some  instructors  thought  that  CT  definitely  transfers  to  the 
combat  environment.  Based  on  their  experience  at  the  National  Training  Center  where  they  had 
observed  success  when  CT  was  used  and  failures  when  it  was  not,  these  instructors  were 
convinced  of  CT’s  efficacy. 

Does  the  guiding  theory,  i.e.  Richard  Paul ’s  conception  of  CT,  provide  adequate  direction 
for  the  development  of  instruction?.  At  the  time  of  the  survey,  CGSC  and  Army  War  College 
courses  were  guided  by  Paul’s  (e.g.  Paul  &  Elder,  2001)  writings  about  critical  thinking.  All 
CGSC  and  Army  War  College  instructors  felt  that  Paul’s  theory  was  adequate.  They  appreciated 
the  normative  standards,  which  permitted  them  to  formulate  feedback  and  ask  questions  within 
class. 


Should  CT  be  taught  in  the  abstract  or  in  the  context  of  the  domain?  Should  CT  be  taught 
covertly  or  overtly?.  Respondents  were  in  strong  disagreement  on  these  topics.  Some  felt  that  CT 
should  be  taught  covertly  because  adult  learners,  particularly  military  officers,  fail  to  see  the 
utility  of  overtly  being  taught  how  to  think  better.  For  these  instructors,  their  student  population 
will  accept  and  benefit  more  from  exercises  that  focus  on  improving  thinking  skills.  However, 
they  will  reject  any  effort  to  directly  teach  those  skills  in  an  abstract  and  overt  way.  It  may  be 
possible  to  come  back  to  the  abstract  principles  after  students  have  experienced  improved  CT 
through  practical  exercises  within  the  domain  of  military  situations.  However,  some  instructors 
felt  that  the  teaching  of  abstract  principles  was  not  necessary.  On  the  other  hand,  some 
instructors  saw  the  overt  and  abstract  principles  as  useful,  although  they  also  thought  it  should  be 
taught  at  more  junior  levels.  It  may  be  that  overt  pedagogy  of  abstract  principles  such  as 
identifying  assumptions,  deductive  logic,  and  making  sound  generalizations  should  be  taught 
early  in  officers’  careers,  when  the  issues  of  adult  learning,  years  of  experience,  and  vast 
amounts  of  existing  knowledge  within  students  are  not  so  robust.  Another  resolution  to  the 
argument  would  be  to  begin  with  covert  instruction,  and  only  at  the  end,  provide  instruction  on 
the  CT  that  students  have  been  engaging. 

What  methods  work  best?  Instructor  opinion  was  unanimous  on  this  question.  All 
instructors  agreed  that  practical  exercises  work  best.  Instructors  did  not  say  that  other  methods 
are  not  effective.  For  example,  practical  exercises  may  do  little  if  not  followed  up  with  feedback 
and  delivery  of  information.  Instructors,  nonetheless,  saw  the  most  improvement  in  their  students 
when  given  exercises  that  allowed  them  to  practice  their  thinking  skills  in  a  hands-on  context. 
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What  successes  have  been  experienced?  Generally,  instructors  felt  evidence  of  their 
successes  came  from  their  students.  Each  reported  that  student  evaluations,  comments,  and 
performance  improvements  were  the  biggest  achievements  they  had  experienced  in  teaching  CT. 
They  reported  improvement  in  student  decision  making,  as  well  as  in  the  quality  of  their 
thinking.  Some  instructors  thought  that  students  were  better  able  to  prepare  for  and  deliver 
briefings. 

What  are  the  biggest  problems?  The  biggest  difficulty  instructors  said  they  faced  comes 
from  the  Anny  itself.  That  is,  the  Army  culture  tends  to  discourage  critical  thinking  to  some 
extent.  Because  of  the  Army’s  hierarchical  nature  and  the  requirement  for  unit  cohesiveness, 
respondents  said  thinking  “out  of  the  box”  is  often  discouraged  by  leaders  and  by  doctrine. 
Officers  are  given  checklists  even  for  the  most  critical  decisions  and  tasks,  such  as  mission 
analysis.  Thus,  the  culture  tends  to  create  a  certain  mistrust  of  critical  thinking  and  creates  a 
corresponding  emphasis  on  decision-making  and  action.  All  of  this  tends  to  decrease  acceptance 
of  instruction  that  focuses  on  improving  critical  thinking.  Elective  courses  in  CT  typically  have 
low  enrollments.  Another  critical  issue  is  that  the  instructors  themselves  need  better  training. 

Conclusions 

In  summary,  instructors  and  curriculum  developers  are  mostly  satisfied  with  how  CT  is 
taught  at  the  schoolhouse.  They  believe  that  it  is  adequately  covered  and  that  the  guiding 
conception  of  CT  (Paul  &  Elder,  2001)  is  useful  and  sufficient.  Their  experiences  suggest  that 
realistic,  practical  exercises  that  require  practicing  CT  are  the  best  way  to  improve  thinking 
skills.  They  also  recognize  that  feedback  about  students’  performance  is  a  necessary  feature  of 
practical  exercises.  They  favor  rich  and  varied  examples  and  realistic  context  for  the  exercises. 
Their  central  complaint  as  teachers  and  curriculum  developers  is  that  the  Army  culture  tends  to 
(1)  create  suspicion  about  slow  deliberate  thought  that  characterizes  CT,  and  (2)  discourage 
innovative  thinking.  For  this  reason,  some  instructors  and  curriculum  developers  avoid  talking 
about  CT  in  the  abstract.  For  example,  they  avoid  instruction  that  refers  to  various  models  of  CT 
and  even  avoid  the  term,  CT.  Instead,  they  attempt  to  encourage  development  of  CT  using  covert 
methods.  Other  respondents  thought  it  was  useful  to  present  and  discuss  models  of  CT, 

The  literature  on  CT  provides  a  different  perspective  of  how  CT  should  be  taught  than  the 
recommendations  from  instructors  based  on  years  of  experience.  These  two  vantage  points  do 
not  necessarily  yield  contradictory  advice  for  new  CT  learning  programs,  however.  Researchers 
and  instructional  experts  agree  that  practical  exercises  are  the  best  way  to  teach  CT  and 
acknowledge  the  possibility  that  distance  based  methods  may  be  effective.  They  also  agree  that 
examples  that  are  realistic,  practical,  and  context-rich  are  effective.  However,  the  literature  has 
more  to  say  about  what  to  train  and  methods  of  training.  While  instructional  experts  report 
satisfaction  with  current  CT  theories  (Paul  &  Elder,  2001)  and  the  use  of  practical  exercises  as  a 
useful  method,  the  literature  suggests  that  other  techniques  may  be  more  effective. 

The  results  of  the  literature  review  and  instructor  survey  informed  the  development  of 
functional  requirements  for  a  distance-learning  program  of  CT  training.  These  requirements  are 
discussed  in  the  next  section. 
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REQUIREMENTS  FOR  WEB-BASED  TRAINING 
OF  CRITICAL  THINKING 


To  lay  the  groundwork  for  designing  an  Internet-based  CT  training  program,  an 
Instructional  Systems  Design  (ISD)  analysis  was  performed.  The  first  steps  in  ISD  include 
specification  of  an  initial  estimate  of  the  capabilities  of  proposed  instruction,  the  scope  of  the 
instructional  problem,  the  location  of  instruction,  anticipated  student  load,  alternative 
instructional  strategies,  support  for  maintaining  the  system,  facilities  requirements,  evaluation 
assessment,  quality  improvement  metrics,  and  funding  requirements. 

In  addition  to  the  literature  review  and  Survey  described  above,  specification  of  the 
functional  requirements  was  partially  addressed  by  conducting  interviews  with  members  of  the 
Intermediate  Level  Education  team  which  was  responsible  for  revising  CGSOC  curriculum  in 
2001.  We  also  observed  lessons  in  CT  given  at  the  School  for  Command  Preparation  (SCP)  at 
CGSC.  The  results  of  our  analysis  are  given  below. 

Functional  Requirements 
Estimate  of  Instructional  Capability 

To  aid  in  estimating  the  proposed  training  system’s  instructional  capability,  its  purpose  and 
format  were  specified.  The  Army  provides  officers  with  a  variety  of  educational  experiences  that 
include  training  in  CT.  Sometimes  that  training  is  embedded  in  courses  on  other  topics  (e.g., 
tactics,  leadership,  etc.),  and  sometimes  CT  is  the  focus  of  the  educational  experience.  Interview 
respondents  suggested  that  what  is  needed  is  courseware  that  (1)  can  be  accessed  by  any  officer 
anywhere  in  the  world,  and  (2)  contains  lessons  apparently  not  addressed  by  current  training.  We 
made  several  assumptions,  supported  by  our  observations  and  interviews,  in  specifying  the 
purpose  of  the  proposed  training  program.  First,  we  assumed  that  Army  officers  currently 
employ  CT  in  the  conduct  of  their  work.  In  other  words,  they  already  have  a  certain  level  of  CT 
skill.  Second,  we  assumed  that  the  existing  CT  training  offered  by  the  Army  supports 
development  of  the  skills  Army  officers  currently  employ.  Third,  we  assumed  that  any  problems 
in  CT  experienced  by  Army  personnel  reflect  deficiencies  that  additional  training  should  target. 
Therefore,  the  following  specifications  regarding  the  purpose  of  the  new  training  were  derived. 
This  new  CT  training  should  be  easily  accessible  on  the  Internet  to  provide  maximal  distribution 
to  Army  personnel.  It  should  complement  and  support  the  Army’s  courseware  for  leadership 
(FM  6-22)  and  other  currently  provided  educational  experiences  and  training  in  CT.  Finally,  it 
should  address  deficits  in  officers’  CT  as  manifested  by  observed  deficiencies  and  errors  in 
human  perfonnance  associated  with  common  battle  command  tasks,  notably  those  that  involve 
the  Military  Decision  Making  Process  (MDMP). 

The  following  specifications  regarding  the  format  of  the  new  training  were  derived.  First, 
the  new  system  should  be  self-directed  WBT  that  is  modularly  organized  around  a  core  .  The 
skills  should  be  important  and  problematic  for  mission  completion.  The  training  should  be 
disseminated  over  the  Web  via  a  user  interface  formatted  within  an  Internet  browser 
environment.  To  encourage  commitment  and  diminish  dropout  rates,  and  ensure  realism  that  will 
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improve  transfer  to  real  world  situations,  it  should  employ  multimedia  presentations  that  include 
charts,  maps,  sketches,  video,  and  animation.  To  ensure  that  it  supports  and  complements 
existing  training,  concepts  based  on  CGSC’s  adopted  Experiential  Learning  Model  (ELM), 
discussed  in  detail  below,  should  be  embedded  in  its  design.  Finally,  to  assist  instructors  who 
might  use  the  system,  it  should  incorporate  performance  and  knowledge  assessment  throughout 
each  module,  with  multiple  branch  points  to  support  individualized  remediation. 

Scope  of  Instructional  Problem 

The  instruction  should  focus  on  a  set  of  critical  thinking  skills  that  are  important  and 
problematic  to  battle  command  and  the  MDMP.  Each  skill  should  be  trained  separately  and 
independent  from  the  rest  to  meet  the  practical  needs  of  instructors  and  students. 

Location(s)  of  Instruction 

The  bulk  of  instruction  should  be  located  on  the  Internet,  accessible  through  an  Army- 
provided  URL.  Some  instruction  may  reside  in  links  to  other  sites  (e.g.,  references,  related  on¬ 
line  courses).  The  system  should  provide  for  a  face-to-face  and/or  e-mail  component,  supplied  by 
an  instructor,  who  may  reside  at  one  of  the  Army’s  schoolhouses.  The  instructor’s  function 
should  involve  answering  questions  about  the  material,  facilitating  deeper  understanding  of  the 
material,  and  remediating  any  misconceptions  of  key  points.  The  web-based  software  should 
reside  on  any  Army-controlled  server  owned  by  any  Army  unit. 

Anticipated  Student  Load 

The  student  target  audience  should  be  any  officer  enrolled  in  one  of  the  Anny’s  training 
and/or  educational  facilities.  Thus,  the  load  may  be  high,  since  it  could  include  both  active  duty 
and  reserve  officers  of  most  ranks  in  the  Army.  Student  load  could  be  even  higher  if  other 
educational  institutions,  such  as  West  Point,  chose  to  employ  the  training. 

Alternative  Instructional  Strategies,  Methods,  And  Media 

Alternative  instructional  strategies,  methods,  and  media  should  be  considered.  The 
instructional  strategy  should  be  designed  to  optimize  student  participation,  student  feedback, 
student  pacing,  and  instructional  sequence. 

Support  for  Developing  and  Maintaining  the  System 

To  develop  the  system,  the  following  resources  are  required.  Access  to  tactical  Subject 
Matter  Experts  and  existing  tactical  and  garrison  scenarios  used  in  current  training  are  necessary. 
Army  participation  in  overseeing/validating  content  development  is  also  required.  To  conduct 
formative  evaluations,  it  will  also  be  necessary  to  obtain  comments  from  students  and/or 
instructors  who  participate  in  pilot  tests  of  the  system. 
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Facilities  Requirements 

Web  hosting  and  maintenance  are  necessary  to  keep  the  training  content  up-to-date.  We 
also  assume  that  the  classroom  facilities  will  be  available  for  delivering  any  face-to-face 
components  of  the  training  that  are  needed. 

Evaluation  and  Assessment 

Access  to  Army  officers  for  formative  assessments  and  to  serve  as  control  and 
experimental  groups  for  outcome  assessments  is  required. 

Quality  Improvement  Metrics 

Assessment  should  be  based  on  development  of  a  measurement  paradigm  and  a  set  of 
measures  of  CT.  The  experience  of  Army  SMEs/instructors  should  be  utilized  for  validation  of 
the  measures.  The  measures  should  assess  how  the  process  of  CT  has  improved  among  the  cadre 
of  officers  taking  the  course.  This  assessment  should  distinguish  CT  process  from  CT  products 
(outputs).  The  assessment  should  be  focused  not  global.  That  is,  it  should  be  designed  to  assess 
the  particular  CTS  that  have  been  addressed  by  the  training,  rather  than  just  CT  as  a  whole. 


Additional  Requirements 

The  interviews  with  the  ILE  team  members  and  the  classroom  observations  yielded  two 
additional  requirements  for  (CT)“.  Both  sets  of  data  suggest  that  special  consideration  should  be 
given  to  the  population  of  adult  learners  for  which  training  would  be  developed.  Second, 
interview  respondents  favored  a  particular  model  of  curriculum  for  adult  learners,  the 
Experiential  Learning  Model  (ELM)  (Kolb,  1984),  which  has  been  adopted  by  CGSC.  Thus,  two 
additional  requirements  for  new  CT  training  are  that  (1)  factors  that  affect  adult  learners  should 
be  considered  in  the  system’s  design,  and  (2)  the  ELM  should  serve  to  guide  curriculum 
development.  Factors  that  affect  adult  learning  are  discussed  first,  followed  by  a  summary  of 
ELM. 

Needs  of  the  Adult  Learner 

As  stated  in  the  functional  requirements,  the  target  audience  for  any  military  instruction  is 
the  adult  learner.  Wlodkowski  (1993)  defines  this  individual  as  one  who  performs  social  roles 
that  our  culture  assigns  to  adults  (e.g.,  citizens,  soldiers),  and  perceives  him  or  herself  to  be 
responsible  for  his/her  life.  In  contrast  to  the  public  educational  system,  which  was  designed  to 
instruct  children,  the  adult  learner  has  a  number  of  needs  that  must  be  addressed  regardless  of 
whether  learning  takes  place  in  a  traditional  classroom  or  a  distance-learning  environment. 

In  particular,  adults  need  to  know  why  they  should  learn  what  is  being  taught.  This  can  be 
achieved  through  engaging  introductory  scenarios  and  realistic  simulated  experiences  early  in  the 
lesson  (Sacks,  1998).  Adults  also  see  themselves  as  self-directing,  responsible  for  their  own 
learning.  Thus,  trainers  should  give  adult  learners  a  choice  of  learning  assignments  to  a  sense  of 
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control.  Since  adult  learners  come  with  experience,  they  will  view  their  lessons  negatively  if  they 
do  not  have  an  opportunity  to  capitalize  on  their  background,  which  will  likely  be  heterogeneous 
across  the  population  of  learners. 

Adults  are  also  highly  pragmatic,  life-centered  learners  who  must  see  the  relevance  and 
immediacy  of  a  task  rather  than  simply  learn  for  the  sake  of  learning.  They  will  also  have  a 
problem-centered  orientation.  They  will  learn  best  from  those  experiences  that  engage  the  senses, 
where  the  most  effective  learning  begins  with  practice  and  ends  with  theory  (Palloff  &  Pratt, 
1999).  Finally,  adults  are  more  responsive  to  internal  motivators  such  as  the  desire  for  self¬ 
esteem.  Yet,  by  the  time  learners  become  adults,  fewer  than  one  in  five  has  a  positive  self-image 
about  their  ability  to  learn  (Rose  &  Nicholl,  1998). 

Within  the  larger  population  of  adult  learners  are  the  demographics  of  younger  students, 
who  belong  to  a  combination  of  Generation  X  (born  between  1965  and  1976)  and  Generation  Y 
(1977-1995).  Military  students  fall  into  these  demographics,  and  because  of  their  heavy 
exposure  to  television  and  other  media,  they  have  high  expectations  for  the  quality  of 
technology-based  instruction  they  receive.  In  this  regard,  Sacks  (1998)  found  that  nearly  half  of 
the  college  students  he  surveyed  chose  entertaimnent  as  the  most  important  characteristic  of  an 
instructor.  Gen-Xers  prefer  fast-paced  presentations  and  want  their  education  combined  with 
entertainment.  They  also  want  to  receive  frequent  feedback,  experience  a  daily  sense  of 
accomplishment,  and  have  freedom  in  choosing  their  assigmnents  and  location  where  they  study 
(Abell,  2000).  In  sum,  the  military  instructional  developer  must  consider  a  variety  of  factors 
when  designing  a  distance  learning  curriculum,  knowing  that  his/her  Gen-X  and  Gen-Y  adult 
learner  students  have  special  requirements. 

For  the  reasons  noted  above,  an  additional  functional  requirement  for  new  distance-based 
training  in  CT  is  that  it  considers  the  special  needs  and  characteristics  of  the  adult  learning 
population. 

Experiential  Learning  Model  (ELM) 

Our  interviewees  said  that,  in  recognition  of  their  adult  student  base,  SCP  and  CGSC  have 
adopted  the  ELM  (Kolb,  1984)  to  provide  overall  guidance  for  the  development  of  their 
instructional  materials,  particularly  lesson  plans.  There  are  several  reasons  why  the  development 
of  (CT)2  should  also  use  this  model.  First,  instructional  developers,  instructors,  and  students  are 
familiar  with  the  model.  Familiarity  is  likely  to  increase  comprehension  and  encourage 
acceptance  of  new  training  materials.  Second,  the  ELM  is  similar  to  instructional  models  that 
have  been  empirically  shown  to  be  effective  for  adult  learners.  Interviewees  who  used  the  ELM 
testified  to  its  effectiveness.  Thus,  it  appears  to  be  a  sound  method  that  is  likely  to  increase  the 
effectiveness  of  new  instruction.  Third,  the  ELM  includes  features  that  are  likely  to  increase 
student  motivation,  which  is  essential  to  any  distance-based  instruction.  For  these  reasons,  use  of 
the  ELM  is  considered  a  final  functional  requirement  for  new  (CT)  . 

The  six-step  model,  depicted  in  Figure  1,  deviates  from  traditional  methods  of  instruction 
by  switching  the  order  in  which  academic  and  practical  infonnation  is  presented.  As  shown  in 
the  figure,  instruction  begins  with  a  concrete  exercise  designed  to  start  everyone  at  the  same 
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point.  Next,  intensive  performance  feedback  is  given  to  the  student,  where  feedback  comes  from 
a  mix  of  peers,  instructors,  and  SMEs.  The  feedback  is  administered  according  to  advanced 
instructional  principles  and  it  is  where  most  learning  is  assumed  to  occur, 


Figure  1.  The  six-step  Experiential  Learning  Model  (adapted  from  Kolb,  1984). 

In  Step  3,  the  students  receive  the  bulk  of  their  academic  instruction,  in  which  the 
principles  underlying  the  task  content  of  the  exercises  are  presented.  Abstract  conceptualization 
is  emphasized  here,  along  with  new  information,  theory,  and  techniques.  During  Step  4,  students 
engage  in  a  second  round  of  concrete  exercises,  in  which  there  are  opportunities  for  active 
experimentation  with  alternative  methods  of  implementing  the  principles  just  presented.  In 
Step  5,  students  demonstrate  competence  by  performing  a  practical  exercise  where  formal  grades 
and  critiques  are  provided.  Finally,  the  students  encounter  new  situations  to  which  these  skills 
should  transfer.  Our  Survey  results  suggest  the  ELM  is  presently  receiving  widespread 
recognition  with  the  Army  and,  as  such,  will  be  a  useful  foundation  for  the  curriculum 
development  of  (CT)2. 
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COMPUTERIZED  TRAINING  IN  CRITICAL  THINKING 


In  this  section,  we  describe  a  training  system  designed  to  increase  the  CT  of  Army  officers. 
The  system.  Computerized  Training  of  Critical  Thinking  or  (CT)",”  delivers  distance-based 
training  via  the  Internet. 

The  design  of  (CT)  was  guided  by  the  results  of  a  series  of  studies,  reported  in  this 
document  and  elsewhere  (e.g.,  Fischer,  et  ah,  2008a;  Fischer,  et  ah,  2008b).  The  studies  led  to 
the  identification  of  a  set  of  pedagogical  principles  that  served  as  the  philosophical  basis  of 
(CT)"’s  design.  Practical  considerations  and  the  functional  requirements  previously  discussed 
also  influenced  the  design  of  (CT)2.  A  discussion  of  the  guiding  pedagogical  principles  and  a 
description  of  the  training  system  are  provided  below. 

Pedagogical  Principles 

The  following  paragraphs  describe  each  principle,  how  it  was  derived,  and  the  training 
system  design  characteristics  it  suggests. 

CT  Skills  Can  Be  Learned,  Trained,  and  Transferred 

The  model  of  CT  presented  in  Volume  Two  of  this  report  posits  that  CT  in  general  can  be 
facilitated  and  enhanced  by  developing  specific  skills  and  that  these  skills  can  be  learned  and 
trained.  This  is  a  fundamental  assumption  about  the  nature  of  CT  that  leads  to  other  assumptions. 
First,  current  thinking  about  skill  acquisition  posits  that  it  develops  over  time  under  the  right 
conditions.  While  skill  performance  may  be  affected  by  inherited  traits  or  abilities,  it  is  generally 
assumed  that  everyone  with  normal  cognitive  capacity  can  develop  some  degree  of  proficiency 
in  executing  skills  that  support  CT.  If  these  skills  are  leamable,  then  it  should  be  possible  to 
develop  a  training  system  that  improves  performance.  Second,  research  has  identified  some  of 
the  conditions  that  are  necessary  to  skill  development,  including  practice,  assessment,  and 
feedback.  Third,  the  conditions  of  training  will  affect  the  degree  to  which  performance  of  any 
skill  transfers  to  real  world  situations  (Halpern,  2004). 

Practice  is  Essential.  It  is  commonly  accepted  that  all  skill  development  requires  practice. 
Moreover,  the  idea  that  skills  that  support  CT  can  be  improved  with  conscious,  deliberate, 
practice  is  posited  by  many  theorists  (e.g.,  Paul  &  Elder,  2001).  Said  another  way,  skill 
performance  does  not  improve  with  an  increase  in  content  knowledge  alone.  It  is  one  thing  to 
understand  how  to  create  topspin  on  a  tennis  serve;  it  is  another  to  actually  do  so.  The  latter 
requires  repeated  attempts  to  create  such  a  serve.  Practice  is  critical  to  skill  development  whether 
the  skill  in  question  is  psychomotor  or  cognitive.  For  example,  the  comprehension  that  is 
obtained  by  reading  math  text  only  goes  so  far  in  developing  the  ability  to  perform  math 
calculations  or  applying  the  correct  equation  to  a  calculus  problem.  Practice  at  perfonning 
calculations  or  applying  a  formula  is  essential  to  developing  these  proficiencies.  Therefore,  the 
first  pedagogical  principle  that  guided  the  development  of  (CT)"  was  that  the  system  should 
include  practice  at  performing  CT  skills.  Thus,  each  (CT)2  module  contains  practice  exercises 
that  require  students  to  practice  CT  skills. 
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Feedback  is  Essential.  An  axiom  of  skill  acquisition  is  that  feedback  about  perfonnance  is 
required  (Kluger  &  DeNisi,  1996).  The  learner  cannot  modify  his/her  perfonnance  without 
knowing  whether,  and  what  kind  of,  changes  are  needed.  Simply,  learning  will  not  occur  in  the 
absence  of  feedback.  Moreover,  if  external  feedback  is  not  provided  in  a  learning  situation,  the 
student  will  very  often  supply  his/her  own  feedback,  evaluating  performance  based  on 
idiosyncratic  standards  that  may  or  may  not  be  appropriate.  Psychological  science  also  shows 
that  immediate  feedback  is  more  effective  than  delayed  feedback.  Therefore,  another 
pedagogical  principle  that  was  adopted  was  that  feedback  should  be  provided  to  students 
immediately  following  each  decision  or  action.  Hence,  descriptive  feedback  is  given  for  each 
response  students  make  when  using  (CT)2  . 

Assessment  is  Essential.  It  is  impossible  to  include  feedback  in  the  absence  of  assessment. 
The  evaluation  of  student  perfonnance  during  training  is  essential  so  that  informative  feedback 
can  be  provided  as  students  practice  a  skill.  In  (CT)2,  students’  knowledge  and  skill 
development  are  assessed  following  each  major  lesson  of  the  training  to  give  them,  and  their 
instructors,  an  understanding  of  where  they  stand  in  the  learning  curve.  The  assessment  measures 
can  also  be  used  before  taking  any  portion  of  the  training  to  determine  whether  or  not  the 
training  is  appropriate  for  the  student. 

Training  Conditions  Should  Optimize  Transfer.  Research  has  shown  that  leading 
students  toward  perfect  perfonnance  in  the  context  of  training  does  not  ensure  that  that 
performance  quality  will  transfer  to  real-world  situations.  Although  the  findings  are  non- 
intuitive,  transfer  is  decreased  by  mastery  learning  requirements  in  training.  Perhaps  perfected 
skills  do  not  transfer  well  to  real  world  applications  because  situational  conditions  in  the 
classroom  never  match  the  complexity  of  real-world  situational  variables.  Thus,  mastery  learning 
conditions  lead  to  the  development  of  inflexible  skills  that  cannot  be  applied  to  unfamiliar 
situations.  If  learners  practice  skills  in  a  variety  of  contexts,  those  skills  are  more  apt  to 
generalize  to  unfamiliar  contexts  (Halpern,  2004).  (CT)"  was  designed  to  optimize  transfer,  as 
opposed  to  optimizing  performance  in  the  context  of  training. 

Part-Training  Methods  are  Most  Effective 

Well-honed  CT  is  essential  to  good  performance  in  many  kinds  of  battle  command  tasks 
and  situations  (Fischer,  et  ah,  2008a).  However,  to  organize  a  CT  training  program  around  such 
real-world  tasks  or  situations  would  be  inefficient  because  a  considerable  amount  of  time  would 
be  spent  on  issues  unrelated  to  CT.  A  training  system  that  taught  CT  by  having  students  perform 
large-scale  battle  command  tasks  would  also  be  less  effective  at  developing  CT  because  the 
focus  of  the  system  would  be  too  broad.  Therefore,  (CT)  provides  students  with  instruction  and 
practice  on  only  those  parts  of  real  world  battle  command  tasks  that  demand  CT  resources,  but 
not  the  whole  task.  (CT")  focuses  on  a  set  of  important  component,  critical  thinking  skills.  It  was 
necessary  to  adopt  a  reasoned  strategy  for  determining  which  critical  thinking  skills  would  be  the 
focus  of  (CT)2.  Two  pedagogical  principles  were  adopted  to  create  the  desired  focus  of  the 
training.  The  first  principle  identifies  the  kinds  of  critical  thinking  skills  on  which  the  training 
should  focus.  The  second  identifies  the  subtasks  that  should  be  emphasized  by  the  training. 
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Focus  on  important  and  problematic  critical  thinking  skills.  Some  training  programs, 
both  military  and  civilian,  attempt  to  teach  CT  in  the  context  of  critical  thinking  skills  that  have 
been  haphazardly  selected.  Instructors  or  curriculum  developers  may  fail  to  systematically 
identify  the  critical  thinking  skills  that  would  benefit  most  from  CT  training.  For  example,  CTS 
may  be  selected  because  they  are  discussed  as  critical  components  of  CT  and  are  recommended 
in  popular  philosophy  books  and  texts.  Without  a  reasoned,  systematic  selection  method,  the 
effectiveness  of  the  resulting  training  program  is  questionable.  There  are  many  critical  thinking 
skills  on  which  the  training  might  be  focused  (See  Appendix  A  for  a  detailed  list).  Moreover,  a 
reasonable  assumption  is  that  Anny  officers  already  perform  many  CTS  well.  To  assume  that 
officers  need  to  be  trained  on  all  possible  tasks  is  a  mistake.  Therefore,  another  pedagogical 
principle  that  guided  system  design  was  that  training  should  focus  on  development  of  critical 
thinking  skills  that  have  been  documented  to  be  important  and  problematic  to  battle  command. 

Research,  reported  in  Volume  Two  of  this  series  and  shown  in  Appendix  C,  identified 
eight  critical  thinking  skills  that  meet  these  requirements.  The  prototype  (CT)  ,  described  in  this 
volume,  focused  on,  and  was  organized  around,  two  of  the  eight  critical  thinking  skills 
previously  identified  as  important  and  problematic  to  Anny  battle  command. 

Focus  on  common  and  consequential  errors.  The  perfonnance  of  complex  critical 
thinking  skills  can  be  broken  down  into  sub-skills,  just  as  physical  tasks  can.  For  example,  when 
serving,  tennis  players  must  be  able  to  position  themselves  at  the  right  place  on  the  court,  grasp 
the  ball  and  racket,  position  the  racket  correctly  in  their  hand,  toss  the  ball  to  the  right  height  and 
location,  position  the  racket  behind  their  arched  back,  swing  to  make  contact  with  the  ball,  and 
follow  through  with  the  racket  face  angled  in  the  right  direction  to  create  topspin.  Similarly, 
critical  thinking  skills,  such  as  Framing  the  Message,  require  the  ability  to  read  or  hear  the 
message,  understand  the  meaning  of  each  word,  comprehend  each  sentence,  relate  each  sentence 
to  the  other  sentences,  and  form  a  mental  structure  that  represents  the  message.  Some  subtasks 
serve  the  purpose  of  supporting  others,  which  may  be  referred  to  as  enabling  subtasks. 

Training  enabling  subtasks  of  a  complex  critical  thinking  skill  would  be  inefficient  and 
unnecessary,  as  it  is  likely  that  students  are  already  perform  them  well.  For  example,  it  would  be 
a  waste  of  time  to  teach  reading  skills  to  Anny  officers  in  an  attempt  to  improve  their  ability  to 
Frame  the  Message.  Moreover,  enabling  subtasks  typically  do  not  greatly  contribute  to  overall 
performance  of  a  skill.  They  may  be  performed  well  by  all  members  of  the  target  population,  or 
they  may  simply  not  be  very  important  to  overall  performance  of  the  task.  In  summary,  training 
unimportant  subtasks  is  a  waste  of  everyone’s  time  and  is  costly. 

If  we  assume  that  it  is  unnecessary  and  inefficient  to  teach  enabling  subtasks,  then  on  what 
subtasks  should  the  training  focus?  One  answer  is  to  detennine  where  training  is  most  needed, 
and  then  focus  training  on  those  problematic  areas.  One  clear  indicator  of  the  need  for  training  is 
error,  especially  if  that  error  is  common  and  if  the  consequences  of  that  error  are  severe. 
Investigations  reported  in  Volume  Two  of  this  report  were  helpful  in  identifying  errors 
commonly  made  and  associated  with  particular  critical  thinking  skills  important  and  problematic 
to  Army  battle  command.  Hence,  the  second  pedagogical  principle  adopted  to  guide  design  of 
(CT)2  was  that  the  training  should  focus  on  those  critical  thinking  skills  and  subtasks  that 
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commonly  produce  consequential  errors.  Therefore,  the  content  of  (CT)  addressed  those  errors 
that  Army  officers  had  reported  as  common  to  battle  command  tasks  they  have  to  perform. 

Use  Training  Methods  Appropriate  for  Adults  (Army  Officers) 

As  stated  in  the  functional  requirements,  the  training  should  target  the  adult  population  of 
Army  officers.  Training  developed  for  adults  should  take  into  account  the  unique  characteristics 
and  needs  of  the  population.  The  relevance  and  importance  of  the  training  should  be  apparent. 
Adult  students  should  be  informed  of  why  they  should  learn  what  is  being  taught.  Because  adults 
are  accustomed  to  directing  their  own  activity,  they  should  be  given  control  over  their  own 
learning  whenever  possible.  Adults’  motivation  will  be  highly  affected  by  factors  that  influence 
their  self-esteem.  Thus,  feedback  is  particularly  important.  To  address  the  needs  of  the  adult 
student  population,  several  pedagogical  principles  were  adopted  to  guide  development  of  (CT)  . 

Use  concrete  experience  to  start.  The  adult  learner  needs  to  be  convinced  at  the  outset  of 
the  training  that  it  is  worth  the  effort  to  participate.  Ensuring  their  engagement  in  the  training  can 
be  accomplished  in  many  ways.  The  ELM,  (Kolb,  1984),,  which  is  used  by  several  military 
training  programs,  employs  a  concrete  experience  in  the  beginning  of  the  training  to  bring  the 
adult  student  to  the  table,  peak  interest,  and  convince  the  student  that  the  training  may  be  useful. 
The  effectiveness  of  the  concrete  experience,  however,  varies  with  the  particular  experience  and 
course.  It  is  necessary  to  make  the  concrete  experience  highly  relevant  and  engaging.  Hence,  one 

principle  that  was  used  throughout  development  of  (CT)2  was  that  a  concrete  experience  should 
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be  given  at  the  beginning  of  every  major  section  of  the  training.  Each  major  lesson  of  (CT) 
includes  a  concrete  experience  in  the  form  of  a  realistic  and  engaging  scenario  that  describes  a 
particular  error  that  is  commonly  made  and  is  associated  with  the  critical  thinking  skill  or 
subtask  being  taught. 

Base  the  training  on  realistic  scenarios.  To  convince  adult  students  that  the  training  is 
relevant  and  important  to  their  work,  it  is  important  to  explicitly  relate  the  training  to  a  familiar 
context.  Realistic  scenarios  are  one  way  to  put  the  adult  in  a  familiar  situation,  task,  and  problem 
and  can  be  highly  effective  in  convincing  adult  students  that  the  training  is  worth  their  while.  In 
addition,  it  greatly  increases  the  likelihood  that  these  skills  will  later  transfer  to,  or  be  applied,  in 
the  real  world.  Moreover,  the  CT  model  discussed  in  Volume  Two  of  this  report  states  that  CT  is 
stimulus-driven.  According  to  the  model,  it  cannot  occur  in  the  absence  of  real-world  stimuli  in 
which  CT  can  and  should  be  applied.  The  richer  and  more  realistic  the  scenario,  the  more  likely 
students  are  to  be  engaged  and  to  buy  into  the  exercises.  Thus,  another  pedagogical  principle 
adopted  was  that  the  training  should  provide  multiple,  rich  scenarios  that  contain  problems  for 
which  CT  can  be  applied.  (CT)"  is  organized  around  such  scenarios. 

Make  training  increasingly  complex  and  difficult.  To  maintain  the  motivation  of  the 
adult  learner  throughout  the  training,  it  is  important  to  keep  him/her  challenged.  Instruction 
should  begin  at  a  level  at  which  the  student  can  successfully  build  and  reinforce  key  foundational 
skills,  then  move  on  to  refine  and  build  upon  those  skills  so  they  can  be  applied  in  increasingly 
more  complex  situations.  The  real  world  requires  the  execution  of  many  complex  critical 
thinking  skills  that  are  associated  with  common  errors.  Additional  instruction  is  needed  only  for 
complex  critical  thinking  skills.  Hence,  the  following  pedagogical  principle  was  adopted.  While 
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the  training  can  begin  with  simple  scenarios  and  exercises,  the  complexity  and  difficulty  must 
increase  as  the  student  progresses  through  the  program.  Scenarios  should  grow  in  complexity 
throughout  the  curriculum,  reflecting  the  expected  increases  in  skilled  performance. 

Training  should  be  Distance-based 

As  stated  previously,  the  training  was  designed  to  be  widely  distributed  via  the  Internet. 
Distance  learning  poses  additional  challenges  to  the  curriculum  developer,  who  must  take  into 
account  problems  with  maintaining  student  motivation.  The  biggest  factor  that  determines 
success  in  distance  learning  courses  is  student  motivation.  Thus,  it  is  important  to  include 
motivation-enhancing  elements  in  any  distance-based  training,  especially  when  that  training  is 
CBT  or  WBT.  Four  pedagogical  principles  were  identified  to  ensure  that  the  training  would 
encourage  motivation. 

Multimedia  is  essential.  If  CT  could  be  readily  obtained  from  pure  reading,  the  best 
media  for  delivering  that  training  would  be  textbooks,  which  are  portable  and  easier  on  the  eyes 
than  computer  screens.  Moreover,  in  the  real  world,  CT  is  applied  to  many  different  forms  of 
material  such  as  graphics,  text,  audio,  film,  direct  observation,  tables,  etc.  Hence,  a  variety  of 
instructional  media  and  formats  should  be  used  in  the  training.  Computers  offer  functionality  that 
cannot  be  obtained  from  textbooks.  Specifically,  computers  offer  multimedia  capabilities  that 
include  audio,  film,  animation,  and  graphics.  Furthermore,  presenting  material  in  a  variety  of 
formats  also  tends  to  encourage  motivation  (Mayer,  2001).  Thus,  the  training  should  capitalize 
on  the  multimedia  capabilities  that  WBT  offers. 

While  acknowledging  the  importance  of  multimedia  presentation,  it  is  important  that  the 
primary  purpose  of  the  proposed  training  be  guided  by  pedagogical  principles  and  not  the  “bells 
and  whistles”  of  the  latest  technology.  While  multimedia  will  tend  to  make  the  training  more 
usable,  it  should  never  lead  the  training  away  from  its  central  purpose,  which  is  to  further 
develop  CT  skills.  Therefore,  (CT)2  was  designed  to  balance  the  motivational  and  entertaining 
benefits  of  multimedia  technology  with  its  driving  purpose. 

Interactivity  is  essential.  Requiring  students  to  actively  interact  with  the  program,  rather 
than  merely  read  through  infonnation  on  a  computer  screen,  will  also  increase  motivation  and 
decrease  dropout  rates  (Mayer,  2001).  While  physical  interactivity  on  WBT  may  be  limited  to 
clicking,  typing  text,  scrolling,  and  dragging  and  dropping,  cognitive  interactivity  is  limitless. 
Hence,  to  the  degree  possible,  interactivity  should  be  an  integral  component  of  the  WBT  in  CT. 
(CT)2  includes  interactivity  that  supports  the  overall  objective  of  increasing  CT. 

Use  scoring.  Adults  respond  well  to  and  are  highly  motivated  by  feedback  about  their 
performance.  (Kluger  &  DeNisi,  1996).  One  of  the  clearest  ways  to  provide  feedback  is  to 
provide  a  score.  In  this  case,  points  are  earned  and  lost  based  on  perfonnance.  Therefore,  another 
motivating  tool  that  should  be  included  is  scoring  of  performance.  However,  both  interactivity 
and  scoring  should  not  be  overdone  such  that  the  training  becomes  video  game-like.  (CT)2 
provides  scoring  when  appropriate  under  the  condition  that  the  scoring  supports  the  overall 
objective  of  increasing  CT. 
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Use  innovative  training  techniques.  Research  suggests  that  one  of  the  best  means  to 
increase  CT  capability  is  the  application  of  one-on-one  Socratic  training.  The  effective 
component  of  the  Socratic  method  is  the  unveiling  of  the  student’s  invalid  assumptions, 
deductions,  inferences,  and  generally,  reasoning.  Distance  learning,  specifically  WBT,  makes  the 
ideal  of  one-on-one  Socratic  training  impossible  to  achieve  with  existing  technology.  It  may, 
however,  be  possible  to  provide  WBT  that  also  targets  the  student’s  poor  reasoning.  To  do  so 
requires  innovative  training  techniques  where  students  perform  exercises  requiring  that  they 
critically  think  and  reveal  their  reasoning.  As  previously  discussed,  feedback  pertinent  to  each 
critical  action  is  necessary,  as  previously  discussed.  Therefore,  the  training  should  include 
innovative  training  techniques  that  require  students  to  apply  CT  to  a  critical  thinking  skill  and 
then  receive  feedback.  (CT)  was  designed  around  innovative  training  techniques  that  permit  the 
provision  of  feedback. 


Course  Overview 

The  web-based  prototype  (CT)  trains  two  CTS  modules:  CTS  1 .  Frame  the  Message  and 
CTS  2.  Identify  the  Gist  in  a  Message.  A  description  of  training  for  these  two  modules  is  given  in 
this  section. 

Course  Objectives 

The  focus  of  the  (CT)"  training  program  is  to  improve  key  skills  that  support  critical 
thinking  and  thus  help  Army  personnel  process  information  more  effectively  and  efficiently. 
Critical  thinking  involves  a  deliberate,  systematic  awareness  of  the  process  and  products  of  one’s 
own  thinking.  This  is  especially  important  in  a  military  setting  where  personnel  are  required  to 
interpret,  analyze,  integrate,  evaluate,  and  remember  infonnation  that  comes  from  a  variety  of 
sources  and  in  a  variety  of  formats,  such  as  text,  audio  transmissions,  diagrams,  tables,  pictures, 
video,  and  so  on.  This  training  program  focuses  on  targeting  common  -  and  potentially  serious  - 
errors  that  people  make  when  they  don’t  apply  appropriate  critical  thinking  skills.  These  errors 
include  overlooking  important  details,  misinterpreting  information,  and  making  incorrect 
assumptions  -  all  of  which  can  lead  to  poor  decision-making.  The  training  program  highlights 
awareness  of  these  errors  and  teaches  specific  techniques  that  can  help  people  overcome  them.  It 
presents  the  student  with  real-world  situations  and  asks  them  to  complete  numerous  thinking 
exercises  that  require  the  practice  and  application  of  CT  in  a  variety  of  realistic  settings. 

For  CTS  1,  by  the  end  of  the  training  program  students  should  be  able  to:  (1)  identify  key 
components  of  a  message  and  the  relationship  between  them;  (2)  generate  appropriate  inferences 
and  recognize  inappropriate  ones;  (3)  identify  weak  (i.e.,  unclear  or  questionable)  parts  of  a 
message  and  be  able  to  apply  the  appropriate  steps  to  resolve  these  weaknesses;  and  (4)  use 
techniques  to  help  them  better  remember,  analyze,  and  evaluate  key  information.  Objectives  for 
CTS  2  are  similarly  specified. 

Target  Student  Population 

(CT)2  was  designed  for  Army  personnel  in  leadership  positions,  although  all  personnel 
could  benefit  from  the  training.  In  order  to  make  the  training  as  meaningful  as  possible,  realistic 
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Army  messages  and  situations  were  employed;  therefore,  before  beginning  the  training  program, 
students  should  have  some  Anny  training  and  experience.  They  should  be  familiar  with  Army 
tenninology,  acronyms,  and  standard  report  formats.  They  should,  for  example,  be  able  to 
understand  standard  Army  messages  such  as  task  organization  tables,  operation  orders,  mission 
statements,  and  battlefield  sketches.  (CT)  assumes  students  already  have  some  domain 
knowledge;  it  gives  them  practice  in  applying  strategies  to  use  that  knowledge  more  effectively. 

Overview  of  (CT)2  Format 

As  Figure  2  illustrates,  (CT)"  is  divided  into  three  sections:  an  introduction  to  the  general 
topic  of  CT,  followed  by  two  main  modules  that  focus  on  particular  critical  thinking  skills.  The 
Introduction  highlights  the  importance  of  developing  critical  thinking  skills  and  outlines  the 
organization  and  methods  used  in  the  course.  The  first  module  focuses  on  the  critical  thinking 
skill  of  framing  a  message,  and  the  second  focuses  on  extracting  the  main  gist  of  messages. 

Each  module  comprises  a  series  of  elements  or  lessons  designed  to  help  students  develop 
specific  techniques  that  will  help  them  interpret  and  critically  perform  the  critical  thinking  skill 
in  question  more  effectively.  Specifically,  each  element  provides  training  that  is  designed  to 
reduce  one  or  more  common  errors  associated  with  the  critical  thinking  skill.  The  elements 
within  each  module  build  on  each  other,  beginning  with  foundational  concepts  and  moving  on  to 
increasingly  more  complex  and  sophisticated  material.  This  ensures  that  students  are  successfully 
able  to  master,  apply,  and  refine  a  skill  before  advancing  to  the  next  one. 


Recommended 

Sequence 


▼ 
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Figure  2.  (CT)2  course  format. 

Module  Format 

Each  module  is  self-contained,  and  presents  training  pertinent  to  a  single  particular  critical 
thinking  skill.  Although  we  strongly  recommend  that  students  progress  through  the  modules  in 
the  suggested  sequence,  it  is  possible  for  an  instructor  to  assign  students  to  individual  modules 
only  or  to  modules  in  a  different  sequence  than  is  recommended.  The  material  and  exercises 
within  a  particular  module  do  not  assume  that  the  student  has  completed  prior  modules.  Each 
module  focuses  on  errors  associated  with  a  particular  CTS. 

Organization  and  Content  of  Modules 

Each  module  begins  with  an  introduction  to  the  particular  critical  thinking  skill  targeted  by 
the  module.  The  introduction  section  of  the  module  opens  with  a  concrete  scenario  illustrating 
the  importance  of  the  task  and  the  errors  that  can  result  when  it  is  not  perfonned  well.  It  then 
goes  on  to  provide  a  clear  definition  of  the  critical  thinking  skill,  as  well  as  key  concepts  and 
terms  and  a  brief  explanation  of  how  this  cognitive  task  relates  to  CT.  This  is  followed  by 
concrete  examples  illustrating  the  key  concepts.  The  introduction  section  concludes  with  an 
outline  of  the  main  objectives  of  the  module  and  a  plan  for  how  the  objectives  will  be  trained. 

The  next  section  of  each  module  consists  of  two  to  five  sub-modules  called  Cognitive 
Elements.  Each  Element  targets  a  common  error  associated  with  the  critical  thinking  skill. 

Element  Format 

Each  of  the  elements  targets  a  specific  CT  error  and  focuses  on  techniques  that  one  can  use 
to  develop  awareness  of  where,  when,  how,  and  why  critical  thinking  skills  should  be  applied. 

As  shown  in  Figure  2,  each  element  contains  the  following  sections: 

1 .  An  opening  scenario  that  illustrates  the  importance  and  prevalence  of  the 
CT  error  covered  in  the  element 

2.  Main  Purpose  of  the  element 

3.  Specific  Lesson  Objectives 

4.  Instruction  providing  content  infonnation,  rationale,  and  practical 
applications 

5.  Training  Examples 

6.  Practice  Exercises 

7.  Summary  of  main  point  of  the  lesson,  practical  considerations  and 
applications 

8.  Exit  Tests 

This  fonnat  is  consistent  with  well-established  lesson  design  principles,  such  as  those  used 
in  the  Experiential  Learning  Model  and  Direct  Instruction  Model  (also  known  as  the  Madeline 
Hunter  Method  (Hunter,  1984)).  The  elements  are  based  on  clearly  specified  objectives  that 
outline  the  purpose  and  goals  of  the  exercises.  The  scenarios  that  begin  each  element  provide  an 
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anticipatory  set,  or  “hook,”  that  not  only  grabs  the  students’  attention,  but  also  gets  them  ready  to 
learn  the  material  by  providing  them  with  a  concrete  situation  that  they  can  relate  to  and  use  to 
organize  the  infonnation.  Many  of  the  scenarios  also  provide  a  means  of  confronting  students 
with  their  own  biases  and  tendencies  to  make  the  errors  that  are  targeted  in  the  lesson — again, 
making  the  material  more  relevant  and  motivating  for  the  students. 

The  instruction  following  the  scenarios  is  designed  to  encourage  students  to  develop  their 
own  meaningful  understanding  of  the  material  through  concise  explanations  interspersed  with 
numerous  examples  and  practice  exercises.  The  examples  used  mainly  involve  a  variety  of 
military  scenarios,  in  order  to  promote  transfer.  The  exercise  format  generally  takes  a  scaffold 
approach,  with  more  structured  guidance  during  the  initial  exercises  and  less  so  during  the  latter 
ones.  The  exercises  themselves  involve  as  much  interactivity  as  possible,  requiring  students  to 
actively  apply  the  skills  and  techniques  they  just  learned.  This  approach  requires  students  to 
engage  with  the  material,  connect  it  to  their  prior  knowledge,  and  to  make  sense  of  it.  Each 
exercise  also  provides  as  much  feedback  as  the  web-based  format  allows.  Some  exercises 
provide  immediate  direct  feedback  (e.g.,  “correct”  or  “wrong”)  while  others  have  students 
compare  their  responses  to  an  expert’s.  Many  of  the  exercises  provide  practice  for  the  exit  tests 
given  at  the  end  of  each  element. 

Assessment  Format 

The  tests  at  the  end  of  each  element  provide  reinforcement  of  the  material  and  inform  the 
student  and  the  instructor  about  the  extent  to  which  the  student  achieved  the  element  lesson 
objectives.  The  series  of  tests  generally  follow  the  format  outlined  in  Table  1. 

Table  1.  End  of  Element  Exit  Tests 


Test 

Purpose 

Question  Type 

Knowledge  Test 

Designed  to  measure  how  well 
students  learned  concepts  and 
terms 

Consists  of  multiple  choice 
questions  on  content 
covered  in  the  element 

Knowledge  Application  Test 

Designed  to  measure  how  well 
students  can  apply  the  concepts 
and  techniques  learned  in  the 
element 

Consists  of  a  brief  scenario, 
message,  or  other  stimulus 
followed  by  multiple  choice 
questions 

Skill  Application  Test 

Designed  to  test  how  well 
students  can  apply  skills  and 
concepts  in  a  realistic  context 

Generally  follows  the  same 
type  and  format  as  the 
training  exercises,  but  is 
more  challenging 

All  tests  are  designed  using  the  best  practices  of  test  construction,  and  are  graded  by  the 
computer — that  is,  they  do  not  require  the  instructor  to  do  the  grading. 
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Access  to  Course 


Students  are  assigned  a  password  from  the  course  administrator.  Only  students  who  have  a 
password  are  able  to  log  on  to  the  program.  The  program  keeps  track  of  the  last  page  students 
accessed  and  brings  them  to  that  page  when  they  next  log  on.  It  also  records  responses  to  test 
items  and  makes  the  responses  and  scores  available  to  instructors. 

Course  Syllabus 


Course  Content  and  Organization 

The  course  is  organized  in  the  following  manner. 

Course  Introduction.  The  course  introduction  provides  a  brief  overview  of  CT, 
explaining  what  it  is  and  why  it’s  important,  and  outlining  how  the  course  is  organized. 

Critical  Thinking  Skill  1:  Frame  the  Message.  The  first  module  provides  instruction  on 
the  critical  thinking  skill,  Frame  the  Message.  It  is  designed  to  promote  the  ability  to  identify  the 
essential  elements  of  a  message,  understand  the  relationships  among  them,  and  describe  a  high- 
fidelity  representation  of  the  message.  It  addresses  common  battle  command  problems  such  as 
difficulty  in  establishing  clear  and  accurate  understanding  of  a  commander’s  intent  statement  and 
difficulty  in  conveying  clear  intent.  The  main  learning  objectives  of  this  module  are:  developing 
awareness  of  message  structure  in  order  to  better  interpret,  evaluate,  critique,  and  remember  key 
information;  and  recognizing  potential  biases  in  interpreting  information.  The  Cognitive 
Elements  that  comprise  this  module,  the  likely  errors  addressed  by  each  of  the  Elements,  and  the 
training  methods  that  target  these  errors  are  outlined  in  Table  2. 
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Table  2.  Elements  of  Critical  Thinking  Skill  1:  Frame  the  Message 


Element 

Known  and/or  Likely  Error(s) 

Training  Methods  that  Target  Errors 

#1:  Identify  a 
Frame  for  a 
Structured 
Message 

1 .  Select  wrong  frame  for  message 

2.  Fail  to  place  information  in 
frame  slot 

3.  Incorrectly  place  infonnation  in 
slot 

1 .  Provide  instruction  on  frames  and  messages 

2.  Provide  instruction  and  practice  with  feedback 
in  selecting  frames  for  structured  messages 

3.  Provide  instruction  and  practice  with  feedback 
in  classifying  structured  message  elements  in 
frame  slots 

#2:  Analyze  an 

Unstructured 

Message 

1.  Fail  to  place  ALL  infonnation 
in  slot 

2.  Incorrectly  place  infonnation  in 
slot 

1 .  Provide  instruction  on  a  multi-use  frame  for 
use  with  unstructured  messages 

2.  Provide  instruction  and  practice  in  classifying 
ALL  unstructured  message  elements  in  multi-use 
frame  slots 

#3:  Identify 

Weak  Spots  in  a 
Message 

1 .  Fail  to  identity  or  recognize 
weak  spots 

2.  Fail  to  identity  reason  for,  and 
thus,  cannot  resolve  weakness 

1 .  Provide  instruction  on  6  reasons  why  a 
message  element  may  be  weak 

2.  Provide  instruction  and  practice  in 
distinguishing  between  unclear  and  uncertain 
weak  spots 

3.  Provide  instruction  and  practice  in  identifying 
weak  spots  and  why  they  are  weak 

#4:  Resolve 

Weak  Spots 

1 .  Fail  to  resolve  weak  spots. 

Hence,  frame  contains  missing 
infonnation,  uncertain,  or  unclear 
infonnation 

1 .  Provide  additional  practice  in  identifying 
reasons  for  weakness 

2.  Provide  instruction  and  practice  in  ways  to 
resolve  weak  spots  specific  to  their  source  of 
weakness 

#5:  Critical  Self- 
Assessment 

1 .  Makes  unjustified  inferences  in 
a  number  of  ways.  For  example: 

(a)  Substitute  what  a  message 
should  say  for  what  it  does  say,  (b) 
Interpret  content  of  message  to  fit 
preconceived  notions,  (c)  Believe 
that  own  inferences  are  given  as 
evidence  in  message. 

2.  Fail  to  recognize  probability 
that  inference  is  conect,  or 
incorrect 

1 .  Provide  instruction  on  assumptions  and 
inferences 

2.  Provide  instruction  on  levels  of  justification  for 
inferences 

3.  Provide  instruction  and  practice  in  using  multi¬ 
use  frame  to  reveal  inferences  (e.g.,  assumptions) 
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Critical  Thinking  Skill  2:  Recognize  Gist  in  Material.  The  second  module  provides 
instruction  in  the  CTS,  Recognize  Gist  in  Material.  It  is  designed  to  promote  the  ability  to  sort 
through  the  details  in  a  message  (written,  graphical,  visual,  auditory,  and/or  tabular)  and  extract 
the  gist  therein.  It  addresses  common  battle  command  problems  such  as  filtering  overly-detailed 
operation  orders  in  order  to  write  clear  and  concise  mission  statements  that  capture  the  gist  of  the 
original  message,  and  reducing  the  amount  of  time  that  time -pressed  lower  echelons  have  to 
spend  on  accurately  extracting  the  essence  of  a  mission.  The  main  learning  objectives  of  this 
module  are:  representing  the  essential  points  of  a  message  in  a  clear,  concise  manner;  and 
recognizing  how  the  process  of  distilling  the  main  idea  encourages  deeper  processing  and  can 
lead  to  increased  ability  to  remember,  evaluate,  and  use  the  information.  The  Cognitive  Elements 
that  comprise  this  module,  the  likely  errors  addressed  by  each  of  the  Elements,  and  the  training 
methods  that  target  these  errors  are  outlined  in  Table  3. 

Table  3.  Elements  of  Critical  Thinking  Skill  2:  Recognize  Gist  in  Material 


Element 

Known  and/or  Likely  Error(s) 

Training  Methods  that  Target  Errors 

#1  Extract 
gist  from  a 
message 

1.  Fail  to  eliminate  details  from  central 
meaning  of  message 

1 .  Provide  instruction  on  removing  details 
from  message. 

#2  Find  the 
anchor  points 
in  a  message 

1 .  Select  incorrect  main  points  of  message 

1 .  Provide  instruction  on  anchor  points,  or 
main  points  of  message. 

2.  Provide  instruction  and  practice  on 
selecting  key  anchor  points,  iteratively 
whittling  message  down  from  12  to  8  to  4 
elements  that  best  capture  its  central  meaning. 

Recommended  Sequence  of  Training 

We  strongly  recommend  that  students  progress  through  the  training  in  sequence,  beginning 
with  the  introduction,  moving  on  to  Critical  Thinking  Skill  1 ,  Critical  Thinking  Skill  2,  and  so 
on.  However,  the  course  has  been  designed  so  that  each  critical  thinking  skill  can  function  as  a 
stand-alone  module.  Because  the  elements  within  each  critical  thinking  skill  build  on  each  other 
and  some  elements  refer  to  concepts  and  activities  in  elements  covered  earlier  within  that  critical 
thinking  skill,  students  will  be  required  to  complete  the  elements  within  each  critical  thinking 
skill  in  order. 

Instructor’s  Role 

The  course  is  designed  to  function  as  a  stand-alone  program  and  can  be  completed  with 
minimal  instructor  involvement;  however,  instructors  can  augment  the  material  by  facilitating 
discussions,  providing  additional  real-world  examples,  and  by  providing  additional  activities  to 
reinforce  and/or  supplement  the  material  covered  in  the  course.  Likewise,  the  course  itself  can  be 
used  to  augment  or  supplement  existing  critical  thinking  and  other  related  courses. 
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Suggested  Implementation 


The  course  can  be  used  in  a  classroom  setting,  as  supplemental  exercises  completed 
outside  of  class,  or  it  can  be  used  as  a  distance-learning  course. 

Estimated  Time  to  Complete  the  Course 

Each  element  is  designed  to  take  approximately  1.5  to  2  hours  to  work  through  the  content 
information  and  do  the  practice  training  exercises.  The  Exit  test  is  designed  to  take 
approximately  20  minutes.  Table  4  outlines  the  estimated  time  to  complete  each  prototype 
module. 

The  expanded  version  of  (CT)~  contains  training  modules  for  eight  CTS.  The  standard 
training  module  for  each  CTS  takes  about  two  hours  to  complete.  In  addition,  the  first  four  skills 
each  have  an  extended  training  module  which  takes  about  ten  hours  to  complete  per  skill,  as  do 
the  prototype  training  modules. 


Table  4.  Estimated  Time  to  Complete  Sections  of  the  Prototype  Course 


Section  of  Course 

Approximate  Time 

To  Complete 

Course  Introduction 

.5  hours 

Critical  thinking 
skill  1 

10-13  hours 

Critical  thinking 
skill  2 

4-5  hours 

Total  time 

15- 20  hours 
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EVALUATION  OF  (CT)2 

In  this  section  of  the  report,  we  describe  an  investigation  that  was  conducted  to  evaluate 
the  effectiveness,  usability,  and  student  acceptance  of  the  (CT)"  prototype.  The  objective  of  the 
investigation  was  to  detennine  whether  (CT)  2  effectively  increases  measurable  indicators  of  CT 
compared  to  two  other  learning  conditions.  The  research  also  assessed  participating  students’ 
attitudes  and  subjective  evaluations  of  (CT)2  as  indicators  of  acceptance  and  usability. 

Method 


Participants 

Nineteen  soldiers  (17  males  and  2  females)  from  the  3rd  Brigade  of  the  85th  Reserves 
Training  Division  volunteered  to  participate  in  the  research.  Eligible  members  of  the  Division 
were  made  aware  of  the  opportunity  in  their  monthly  meeting  held  by  the  85th  Reserve  Division. 
Potential  participants  were  given  a  briefing  describing  the  purposes  and  procedure  of  the 
research.  Following  the  briefing,  they  were  requested  to  indicate  their  willingness  to  participate 
by  providing  their  name,  email  address,  and  phone  number  on  a  signup  sheet.  At  that  point, 
volunteers  were  given  an  informed  consent  document  to  read  and  sign  as  well  as  a  brief 
demographic  survey. 

Because  (CT)"  was  designed  for  Army  officers  and  senior  enlisted  personnel,  participation 
in  the  research  was  limited  to  those  individuals  with  a  rank  of  E5  to  06.  Three  Captains,  six 
Majors,  six  Lieutenant  Colonels,  two  Colonels,  one  1st  Sergeant,  and  one  Master  Sergeant 
participated  in  the  research. 

Upon  completing  the  research,  all  participants  received  (1)  a  certificate  of  achievement 
issued  by  their  commanding  officer,  and  (2)  a  $75  check  that  was  made  out  to  the  unit’s  family 
support  group. 

Materials 

Participants  were  randomly  assigned  to  one  of  three  CT  training  conditions.  They  either 
received  web-based  training  in  one  CT  module  of  (CT)2,  completed  distance-based  coursework 
modeled  after  an  existing  CT  class  offered  at  CGSC,  or  received  no  training  beyond  what  they 
had  already  been  given  as  Reserve  Army  officers.  All  participants  were  first  asked  to  complete  a 
demographic  questionnaire  that  gathered  information  about  their  rank,  job  position,  and  gender. 
Different  sets  of  instructional  materials  were  developed  and  delivered  to  participants  depending 
on  the  condition  to  which  they  were  assigned. 

Materials  for  (CT)  group.  Participants  assigned  to  the  (CT)"  group  were  exposed  only  to 
training  materials  provided  by  the  web-based  training.  Specifically,  they  were  instructed  to  read 
the  material  on  the  first  page  of  the  website,  register  as  a  student,  read  the  introduction  to  CT, 
and  work  through  all  the  readings  and  exercises  for  the  first  critical  thinking  skill  covered  by 
(CT y,  Frame  the  Message.  A  description  of  the  material  covered  in  this  critical  thinking  skill  is 
given  in  Section  Four  of  this  report. 
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Materials  for  the  distance-based  simulated  CGSC  group.  The  alternative  training  group 
was  provided  with  written  instructions,  reading  assignments,  and  written  exercises.  The  reading 
assignments  covered  a  description  of  critical  thinking  elements  as  developed  and  described  by 
Paul  and  his  colleagues  (Paul,  et  al.  1990;  Paul,  1995;  Paul  &  Elder,  2001).  The  exercises  asked 
them  to  critically  review  substantive,  controversial  materials  that  were  relevant  to  national 
security  and  therefore  to  the  strategic  environment  in  which  they  operated.  The  critical  reviews 
required  the  participants  to  use  Paul’s  elements  and  standards  of  CT  as  criteria  for  their 
evaluation.  Each  participant  was  provided  feedback  on  his  or  her  responses. 

Assessment  instrument.  To  determine  the  relative  effectiveness  of  the  three  conditions  at 
encouraging  CT,  an  assessment  instrument  was  developed  and  administered  to  all  participants  at 
the  end  of  the  research.  The  assessment  instrument  was  designed  to  measure  the  degree  to  which 
an  individual  used  CT  in  answering  a  series  of  questions  about  four  different  messages. 

The  four  messages  were  selected  for  relevance  and  interest.  The  first  was  a  web  page 
graphic  that  described  the  NCO  Vision  as  provided  by  Sergeant  Major  of  the  Army,  Jack  L. 
Tilley.  Message  #2  was  an  Army  Directive,  a  one-page  letter,  from  the  Secretary  of  the  Army, 
Thomas  E.  White.  The  third  message  was  a  one-page  statement  published  by  Al-Qaeda  on  the 
Internet.  The  final  and  fourth  message  was  a  one-page  extract  of  President  Bush’s  National 
Security  Strategy,  published  a  few  months  after  September  1 1 ,  200 1 . 

Following  each  message  were  three  sets  of  questions,  organized  in  three  corresponding 
sections.  The  first  section  contained  several  questions  that  assessed  participants’  memory  for  the 
material  contained  in  the  message.  The  memory  questions  elicited  the  recall  of  particular 
components  of  the  relevant  message.  Section  Two  contained  several  questions,  each  of  which 
asked  the  participant  to  interpret  the  message.  The  interpretation  questions  asked  for  information 
that  was  not  explicitly  contained  in  the  message,  but  could  be  derived  from  the  message’s 
content.  Participants  were  instructed  to  use  their  experience  and  judgment  to  answer  the 
interpretation  questions.  They  were  also  told  to  justify  their  responses,  and  they  were  allowed  to 
refer  back  to  the  relevant  message.  Section  Three  asked  the  participant  to  evaluate  the  message. 
The  questions  in  the  evaluation  section  asked  participants  to  judge  the  quality  of  the  message, 
whether  the  message  was  effective,  and  whether  there  were  any  parts  of  the  message  that  were 
weak,  illogical,  not  believable,  etc. 

Each  assessment  package  was  labeled  at  the  top  with  a  single  participant  control  number  to 
ensure  confidentiality  of  responses.  The  control  number  ensured  that  participants’  names  could 
not  be  associated  with  the  responses.  The  number  also  afforded  grading  of  the  responses  that  was 
blind  to  the  participants’  group  assignments. 

The  assessment  package  for  the  participants  assigned  to  the  (CT)  training  condition  also 
contained  a  user  acceptance  section.  The  user  acceptance  section  contained  questions  concerning 
participants’  evaluation  of  the  training.  It  elicited  their  general  evaluation  of  (CT)2  as  well  as 
their  reactions  to  the  length  of  time  it  required,  its  effectiveness  at  promoting  learning,  and  its 
relevance  to  their  work.  It  also  elicited  their  recommendations  for  the  web-based  training  system. 


34 


Procedure 


After  control  numbers  were  assigned,  the  participants  were  randomly  assigned  to  the  three 
conditions  of  the  experiment.  This  insured  that  a  sequence  of  numbers  would  not  give  any  clue 
as  to  who  was  assigned  to  which  group. 

Six  individuals  were  instructed  to  access  and  complete  the  training  material  for  one  module 
of  the  web-based  critical  thinking  training  program.  Five  participants  who  were  assigned  to  the 
distance-based  simulated  CGSC  group  were  given  the  readings  and  assignment  materials 
previously  discussed.  The  final  group  of  eight  participants  served  as  a  control  group  and  did  not 
receive  any  additional  training. 

Procedure  for  (CT)2  training  condition.  Participants  assigned  to  the  (CT)2  condition 
were  sent  e-mail  messages  that  contained  instructions,  the  Internet  address  to  (CT)2,  and  a 
password  that  gave  them  access  to  the  training  program.  They  were  instructed  to  go  to  the  given 
Internet  address,  register  as  a  student,  and  begin  the  training  program.  Once  registered  as  a 
student  in  (CT)2,  the  program  directs  all  further  training  and  is  essentially  self-contained.  (CT)2  is 
self-paced  and  self-directed  such  that  very  little  or  no  administrative  activities  are  required  of  an 
instructor  or  administrator.  The  program  incorporates  immediate  feedback  for  each  exercise  it 
contains.  It  also  documents  the  results  of  intermediate  assessments  as  part  of  the  training.  In  this 
investigation,  students’  progress  was  monitored  by  periodically  checking  the  data  files  to  ensure 
that  they  were  completed  in  a  timely  fashion. 

Procedure  for  distance-based  simulated  CGSC  group.  The  training  that  was  modeled 
after  a  CT  course  offered  by  CGSC  that  was  also  distance-based.  However,  this  training  was 
entirely  administered  via  e-mail.  Students  were  given  two  assignments.  The  first  assignment  was 
delivered  with  the  first  e-mail  message  sent  to  participants  in  this  group.  Attached  to  the  email 
were  four  files.  Participants  in  the  alternative  training  condition  were  instructed  to  read  an  essay, 
described  previously,  contained  in  the  first  file.  Then  they  were  asked  to  review  a  quick 
reference  guide.  Third,  they  were  told  to  view  a  PowerPoint  slide  that  contained  a  lesson  on  the 
general  topic  of  CT  consistent  with  Paul’s  and  Elder’s  text  book.  Once  the  student  believed  that 
they  understood  the  materials,  they  were  instructed  to  open  the  last  file,  which  contained 
questions  on  the  essay  and  CT  lecture.  They  were  asked  to  answer  the  questions  in  the  boxes 
provided  in  the  document.  When  they  were  finished  answering  the  questions,  which  constituted 
the  first  exercise,  they  were  told  to  save  the  file,  and  return  it  to  an  experimenter. 


Students  were  sent  to  the  second  exercise  when  they  returned  the  first  assignment,  Two 
folders  were  sent  to  them.  The  first  contained  written  feedback  concerning  their  responses  to  the 
first  assignment.  The  other  contained  information  they  needed  for  the  second  assignment.  They 
were  told  to  find  two  documents  in  the  second  file.  One  document  was,  again,  an  essay  they  were 
asked  to  read  and  evaluate.  The  second  document  was  an  analysis  guide  that  contained  a  series  of 
questions  and  written  assignments.  Students  were  instructed  to  complete  the  analysis  guide  and 
return  it  to  an  experimenter.  Upon  completing  the  second  assignment,  the  alternative  training 
was  finished.  Participants  were  sent  feedback  concerning  their  work  on  assignment  #2.  They 
were  also  informed  that  they  would  next  take  a  test  on  critical  thinking  and  were  admonished  to 
not  discuss  the  training  with  their  fellow  participants.  Through  the  alternative  distance-based 
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training,  participants  were  encouraged  to  ask  questions  and  to  contact  an  experimenter  by  e-mail 
or  by  phone.  The  progress  of  individuals  assigned  to  this  group  was  monitored  to  ensure  that 
they  adequately  completed  the  assignments. 

To  ensure  that  the  control  group  received  a  roughly  equivalent  amount  of  interaction  with 
the  experimenters  during  the  training  period,  periodic  updates  about  the  progress  of  the  research 
were  emailed  to  them. 

Participants  who  were  assigned  to  either  training  condition  had  two  months  to  complete 
their  program.  At  the  end  of  the  prescribed  training  period,  all  three  groups  were  administered 
the  assessment  instrument  described  previously. 

Assessment  procedures.  The  test  was  administered  to  participants  during  one  of  the 
monthly  brigade  meetings.  Following  general  instructions  concerning  the  assessment  procedure, 
participants  were  provided  with  the  first  message  to  critically  review.  They  were  permitted  to 
take  notes  on  the  paper  on  which  the  message  was  printed.  Participants  were  instructed  to  return 
the  message  and  any  notes  made  about  the  message  to  the  test  proctor  when  they  had  finished 
review  of  the  first  message.  The  proctor  then  issued  the  memory,  interpretation,  and  evaluation 
questions  for  that  message.  This  procedure  was  repeated  for  the  remaining  three  messages. 

No  time  limits  were  set  for  completion  of  the  assessment  instrument.  The  fastest  time  that 
a  participant  finished  was  just  under  two  hours.  The  longest  time  was  about  four  hours. 

Results 

The  results  presented  in  this  section  are  organized  by  the  two  broad  questions  the  research 
was  designed  to  answer.  First,  the  research  assessed  user  acceptance  and  usability  by  eliciting 
participants’  attitudes  and  opinions  about  (CT)  .  Thus,  we  first  discuss  the  responses  of  six 
individuals  who  participated  in  the  web-based  training  to  questions  about  course  length,  course 
relevance,  acceptance,  perceived  learning  benefit,  and  recommendations.  The  results  of  a  similar 
user  acceptance  survey  of  the  distance-based  course  modeled  after  the  CGSC  course  are  also 
given  to  afford  a  comparison  of  the  two  educational  methods.  Second,  the  research  assessed  the 
relative  CT  performance  of  the  three  training  groups  to  determine  if  (CT)2  provide  additional 
benefits  not  attainable  by  current  CT  training.  The  second  half  of  the  results  section  describes 
how  well  the  three  groups  did  on  the  final  assessment  instrument. 

User  Acceptance  of  (CT)2 

Length  of  training.  Participants  reported  spending  an  average  of  13  hours  to  complete  the 
single  module  of  (CT)  ,  which  included  the  introductory  sections  to  CT,  the  introduction  to  the 
module,  and  the  module’s  five  elements.  However,  the  time  devoted  to  the  training  varied 
considerably.  Several  participants  completed  the  work  between  10  and  15  hours.  However,  one 
participant  was  able  to  complete  the  training  in  6  hours  while  one  reported  that  it  took  him  18 
hours.  Despite  the  considerable  investment  in  time,  each  also  reported  that  it  was  an  appropriate 
amount  of  time  to  spend;  the  training  was  neither  too  long  nor  too  short. 
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Several  participants  reported  that  the  self-paced  aspect  of  (CT)~  allowed  them  to  work  on 
the  training  over  several  sessions,  usually  lasting  one  to  two  hours.  Because  it  was  self-paced,  it 
did  not  seem  excessively  long.  The  material  was  substantive  and  interesting  to  hold  their 
attention  for  the  time  it  took  to  complete  the  training.  However,  they  also  reported  that  they 
would  not  have  wanted  to  spend  more  than  2  hours  in  any  one  session.  They  uniformly 
appreciated  the  self-paced  feature  of  the  training,  remarking  that  some  sections  required  more 
study  than  others.  The  self-paced  feature  allowed  them  to  review  the  more  difficult  concepts. 

Relevance.  Each  of  the  participants  reported  that  the  material  was  highly  relevant  to  their 
work,  both  in  the  civilian  and  military  sectors.  Several  noted  that  they  had  experienced  situations 
during  and  after  the  training  in  which  they  had  applied  what  they  had  learned.  For  example,  one 
instructor  remarked  that  he  had  used  the  training  to  refine  lectures  and  materials  used  in  his  class. 
Another  reported  that  he  had  used  it  to  analyze  a  multitude  of  email  messages  he  received  in  his 
civilian  work,  and  refine  those  he  sent  as  well.  In  his  particular  civilian  job,  his  coworkers  were 
distributed  throughout  the  world  such  that  issues  were  resolved  in  series  of  messages,  typically  e- 
mail,  but  sometimes  phone.  He  reported  that  the  training  helped  him  to  resolve  several  issues  and 
probably  helped  him  avoid  several  potential  problems.  One  reported  that  the  training  could  be 
applied  to  combat  arms  or  service  support  to  understand  infonnation  that  one  is  given  by  others. 
He  noted  that  everyone  takes  short  cuts  in  understanding  messages,  and  that  the  training  helped 
him  to  avoid  some  of  those  short  cuts.  Another  thought  it  was  especially  relevant  to  the  staff 
environment  where  it  could  be  applied  to  messages  used  in  daily  tasks  and  requirements. 

General  evaluation.  All  participants  reported  that  they  found  the  training  interesting. 
Participants  distinguished  interest  from  enjoyment,  however,  reporting  that  they  found  it 
interesting  but  not  necessarily  enjoyable.  One  thought  it  would  enhance  his  military  and  civilian 
career,  and  for  that  reason  saw  value  in  the  training.  Several  commented  that  it  was 
professionally  interesting.  The  self-paced  aspect  of  the  training  was  reported  to  be  beneficial  in 
maintaining  interest. 

The  audio  features  of  the  training  were  found  to  be  particularly  useful.  Participants  thought 
the  audio  portions  made  them  approach  a  message  a  little  differently  than  it  if  were  written  in 
text  fonn  alone.  They  also  liked  the  realism  provided  by  the  audio  and  thought  it  helped  to  keep 
them  interested  and  focused.  One  commented  that  the  audio  afforded  the  perception  of  the 
speaker’s  emotion,  which  gave  him  a  different  perspective  from  which  he  could  understand  the 
message.  However,  several  participants  wanted  the  option  of  skipping  the  audio  portions  because 
the  large  audio  files  took  a  little  more  time  to  load  than  did  the  simple  text. 

Participants  liked  the  fact  that  the  training  was  incremental  and  each  element  built  on  the 
previous  element.  The  logical  sequence  was  well  received.  The  interactivity  provided  by  the 
training  was  also  appreciated. 

Several  participants  reported  that  the  inability  to  review  material  they  had  previously 
studied  was  a  problem  with  the  training.  Because  the  training  is  self-paced,  they  reported  that  it 
was  sometimes  several  days  before  they  would  get  back  to  the  training.  At  these  points,  they 
would  have  liked  to  review  the  previous  material  before  moving  on.  However,  the  training,  as 
was  designed,  prohibited  use  of  the  web  browser’s  back  button,  and  there  was  no  way  for  users 
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to  review  materials.  Other  participants  remarked  that  within  a  session,  they  wanted  to  go  back 
and  review  materials  but  were  unable  to  do  so. 

Several  participants  who  used  modified  web  browsers  also  experienced  difficulty  with  the 
audio  files  incorporated  into  the  training.  One  participant  experienced  occasional  “hiccups”  when 
loading  pages  and  needed  to  either  re-load  the  page  or  come  back  to  the  training  at  a  later  time. 

The  participants  would  uniformly  recommend  the  training  to  their  colleagues  and 
subordinates.  One  remarked  that  the  training  was  appropriate  for  any  soldier  who  would  be  in  a 
leadership  position.  One  thought  the  training  should  be  provided  early  in  an  officer’s  career. 
Others  thought  that  the  training  was  appropriate  for  all  ranks.  Several  of  the  participants  were 
instructors  in  their  civilian  and  military  jobs.  Those  individuals  thought  they  would  use  the 
training  as  preparation  for  other  training.  As  instructors,  they  would  direct  their  students  to  go  to 
the  web  site  and  complete  a  set  of  elements  in  preparation  for  other  course  work.  They  liked  the 
distance  learning  aspect  of  the  web-based  training  and  would  much  rather  use  web-based 
training  than  training  provided  on  a  CD. 

Self-reported  learning.  Participants  reported  that  they  thought  the  training  had  a  positive 
impact  on  their  thinking  skills.  They  believed  the  training  had  to  have  an  impact  because  it 
identified  and  reinforced  the  need  for  careful  message  framing  and  analysis.  One  thought  it  was 
helpful,  but  like  anything  else,  additional  practice  was  necessary  to  see  the  most  benefit  from  the 
training.  The  skill  must  be  applied  routinely,  or  it  will  not  develop.  Another  reported  that  it  made 
him  realize  that  what  he  writes  or  says  must  be  clear  to  avoid  misunderstandings  from  his 
audience.  One  reported  that  he  learned  the  most  from  the  sections  that  required  him  to  analyze 
messages  into  their  components. 

Recommendations.  Participants  had  several  recommendations  to  improve  the  training. 

The  one  most  frequently  cited  was  to  include  the  ability  to  review  previously  learned  material. 
One  participant  wanted  to  see  a  greater  variety  of  examples  and  others  called  for  more  audio 
messages. 

Participants  said  the  examples  and  explanation  provided  by  the  training  were  usually  clear, 
but  several  portions  would  benefit  from  improved  clarity.  One  participant  thought  that  the 
training  was  aimed  at  the  management  or  business  class.  He  thought  that  the  terminology  and 
written  material  might  be  too  high  for  someone  at  say,  an  NCO’s  level.  Vocabulary  that  is  more 
common  might  improve  the  training. 

The  inclusion  of  scoring  systems  in  some  elements  received  mixed  reviews.  On  the  one 
hand,  participants  thought  that  the  scores  made  them  pay  closer  attention  and  work  harder. 
Scoring  served  as  motivators,  which  was  its  original  purpose.  However,  participants  complained 
about  some  of  the  scores  they  received,  especially  when  the  system  failed  to  give  them  points  for 
an  answer  that  they  considered  correct.  That  is  not  to  say  that  participants  generally  disliked  the 
answers  and  explanations  given  by  the  training.  In  fact,  several  participants  reported  that  they 
learned  from  the  explanations  and  the  opportunity  to  receive  feedback  on  their  choices. 
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One  participant  did  not  like  the  way  some  elements  of  the  training  forced  users  to  choose 
only  correct  answers.  He  thought  a  better  system  would  be  to  allow  them  to  make  mistakes  and 
then  receive  feedback.  The  following  bulleted  items  summarize  the  recommendations  of  the 
participants. 


•  Allow  review  of  previously  studied  content/pages 

•  Include  a  greater  variety  of  examples 

•  Include  more  audio  portions 

•  Clarify  some  of  the  explanations 

•  Simplify  the  vocabulary  and  language  used  in  the  explanations 

•  Consider  eliminating  the  scoring 

•  Allow  participants  to  choose  the  wrong  answer 

Comparison  to  alternate  training  group.  Those  participants  who  received  the  alternate 
training  (based  on  training  provided  at  CGSC)  also  evaluated  their  training.  They  took  an 
average  of  6  hours  to  complete  their  training,  which  some  regarded  as  too  short.  They  thought 
they  needed  more  information  to  better  understand  the  process  of  critical  thinking.  As  a  group, 
these  participants  called  for  more  content.  They  did  believe  it  was  relevant  to  their  study  at  the 
Anny  War  College.  One  pointed  out  that  the  ability  to  perform  logical  reasoning  in  a  standard 
format  ensures  comprehensive  analysis.  They  thought  it  was  good  for  overall  officer 
development.  As  a  group,  they  thought  it  was  moderately  interesting.  One  remark  was  "The 
training  was  okay.  Training  is  training,  you  can’t  really  get  excited  about  it.”  They  would 
recommend  it  and  they  think  it  would  help  their  colleagues. 

Performance  Results 

Scoring.  Three  measurable  indicators  of  CT  were  used  to  score  the  responses  provided  by 
all  participants  on  the  assessment  instrument.  The  three  indicators  constitute  measures  of 
memory,  inference  justification,  and  questions  about  the  validity  or  veracity  of  information.  Each 
indicator  was  derived  from  assumptions  about  CT  in  the  general  CT  literature  and/or  in  the 
model  of  CT  described  in  Volume  Two  of  this  report.  A  memory  indicator  was  included  because 
CT,  by  definition,  involves  greater  depth  of  processing.  Hence,  if  CT  is  engaged,  then  processing 
of  information  will  be  deeper.  The  cognitive  psychological  literature  commonly  accepts  the 
notion  that  memory  for  processed  information  improves  as  that  processing  deepens.  Thus, 
memory  for  material  may  be  used  as  an  indicator  of  CT.  The  failure  to  justify  inferences  derived 
from  the  content  of  a  message  is  an  inverse  indicator  of  CT.  According  to  the  CT  model 
presented  in  Volume  Two  of  this  report,  CT  is  driven  by  System  Two  processing.  If  one  accepts 
the  model’s  proposition  that  System  Two  processing  involves  checks  on  the  products  of  the 
automatic  activity  of  System  One,  then  evidence  that  one  has  failed  to  engage  in  such  checks 
would  be  an  indicator  that  one  has  failed  to  critically  think.  Inferences  derived  from  the  content 
of  a  message  are  natural  products  of  System  One  associational  processes.  However,  failing  to 
justify  those  inferences,  even  after  instruction  to  do  so,  is  evidence  that  System  Two  checks  have 
not  been  applied  to  those  System  One  products.  Similarly,  questioning  the  validity  of  given 
information  in  a  message  is  direct  evidence  that  System  Two  checks  have  been  applied;  hence,  a 
positive  indicator  of  CT. 
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The  responses  to  the  test  questions  that  followed  each  message  in  the  assessment 
instrument  were  scored  for  each  of  the  three  CT  indicators.  First,  the  amount  of  infonnation 
remembered  from  each  of  the  four  messages  in  the  test  was  scored.  As  previously  indicated,  each 
message  was  followed  by  a  series  of  questions  that  asked  participants  to  remember  features  of 
the  message  they  had  just  read.  The  memory  questions  were  scored  for  accuracy,  and  then  the 
resulting  memory  scores  were  summed  across  the  four  messages  to  obtain  a  single  memory  score 
for  each  of  the  19  participants. 

To  score  the  number  of  unjustified  inferences  made  by  each  participant,  all  inferences 
(whether  justified  or  not)  made  in  the  interpretation  section  of  the  assessment  test  were  first 
identified  and  counted.  A  statement  was  judged  to  be  an  inference  if  it  clearly  contained  content 
that  was  not  provided  by  the  relevant  message.  For  example,  several  respondents  indicated  that 
the  A1  Qaeda  author  of  the  third  message  believed  that  the  United  States  blames  Afghani 
Muslims  for  the  attack  on  the  Twin  Towers.  This  is  a  clear  inference  because  no  mention  of  the 
attack  is  made  in  the  statement.  Second,  the  number  of  inferences  for  which  no  justification  was 
provided  was  counted.  Justification  was  defined  as  any  mention  of  information  given  in  the 
message  that  a  participant  related  to,  and  provided  as  evidence  for,  the  inference  they  had  made. 
Participants  were  awarded  a  single  point  for  each  unjustified  inference  they  made.  The  points 
earned  for  each  message  were  then  summed  across  the  four  messages  to  obtain  a  single  score  that 
reflected  the  degree  to  which  participants  made  unjustified  inferences. 

To  determine  the  relative  degree  of  questioning  made  by  the  three  groups,  the  number  of 
times  participants  expressed  a  question  about  the  validity  or  veracity  of  information  was  counted 
for  each  message.  Participants  were  awarded  a  single  point  each  time  they  questioned  or  doubted 
information  given  to  them  in  a  message.  The  questions  contained  in  the  evaluation  section  of  the 
assessment  instrument  were  scored  for  this  variable.  The  points  earned  for  each  message  were 
then  summed  across  the  four  messages  to  obtain  a  single  score  that  reflected  the  degree  to  which 
participants  questioned  information  that  was  given  to  them. 

Experimenters  who  had  not  served  as  test  administrators  completed  the  scoring.  Therefore, 
scorers  had  not  met  nor  interviewed  any  of  the  participants  and  were  blind  to  the  experimental 
condition  of  the  each  participant. 

Memory  for  information.  The  average  number  of  items  of  information  remembered  from 
the  four  messages  for  the  entire  sample  of  individuals  was  22.1  (SD  =  6.4).  The  six  participants 
who  participated  in  the  (CT)“  training  remembered  significantly  more  pieces  of  information  (M  = 
26.5,  SD  =  6.9)  than  did  the  individuals  who  took  the  alternative  distance-based  training  (t(9)  = 
1.971,/?  =  .04,  one  tailed)  (M  =  19.8,  SD  =  4.1)  and  than  the  control  group  (t(  12)  =  1.801,/?  = 

.04,  one  tailed)  (M  =  20.1,  SD  =  5.9).  In  other  words,  the  (CT)2  group  recalled  31.6%  more 
unique  pieces  of  information  than  the  control  group  and  33.8%  more  than  the  other  distance- 
based  training  group. 

Number  of  unjustified  inferences.  The  average  number  of  unjustified  inferences 
produced  by  the  entire  sample  of  participants  was  10.5  (SD  =  7.8).  The  (CT)"  training  group 
produced  significantly  fewer  unjustified  inferences  than  the  alternative  distance-based  training 
group  (f(9)  =  5.056,/?  =  .0005,  one  tailed)  and  the  control  group  (t(  12)  =  1.858,/?  =  .04,  one 
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tailed).  The  web-based  training  group  made  an  average  of  4.3  (SD  =  1.6)  inferences  that  they 
failed  to  justify  with  infonnation  given  in  the  message.  The  alternative  distance-based  training 
produced  nearly  four  times  as  many  unjustified  inferences  (M  =  18.0,  SD  =  6.4).  The  control 
group  made  10.4  unjustified  inferences  on  the  average  (SD  =  7.8). 


Questions  of  belief.  The  three  groups  did  not  differ  with  regard  to  the  number  of  questions 
of  belief  produced. 


Conclusions 
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The  results  of  this  research  indicate  that  military  students  find  (CT)  highly  acceptable. 
Although  the  sample  of  participants  who  used  (CT)2  was  small,  it  was  uniformly  positively  rated. 
Despite  the  extensive  time  commitment  the  training  program  requires,  users  found  it  interesting 
and  well  worth  their  time.  Participants  thought  the  program  offered  training  not  available 
elsewhere  in  the  Army.  The  self-paced  feature  of  the  program  appears  to  be  one  of  the  reasons  it 
received  favorable  reviews.  Users  thought  it  was  highly  relevant  and  beneficial  to  their  military 
and  civilian  work. 

Although  (CT)2  was  well  regarded,  reviewers  expressed  several  recommendations  and  it  is 
clear  that  the  usability  of  the  program  needs  improvement.  Specifically,  the  program  needs  to 
have  greater  ability  to  navigate  to  previous  pages  and  more  flexibility  to  use,  or  not  to  use,  its 
audio  components.  Users  also  wanted  more  flexibility  in  their  responses  to  allow  them  to 
actually  make  errors,  which  the  program  currently  prohibits. 

(CT)"  also  appears  to  be  generally  effective  at  encouraging  critical  thinking,  at  least  about 
messages  Anny  personnel  must  evaluate.  The  research  clearly  showed  that  the  web-based 
training  enhanced  memory  for  messages,  possibly  because  it  encourages  greater  depth  of 
processing  in  a  number  of  ways.  For  example,  the  five  modules  studied  by  the  participants  were 
designed  to  sensitize  students  to  errors  they  are  likely  to  make  when  encoding  a  message.  It 
teaches  students  to  examine  the  entire  message,  attempting  to  dissuade  them  from  overlooking 
any  piece  of  infonnation.  It  also  teaches  them  to  carefully  examine  weak  components  of  a 
message  and  attempt  to  clarify  their  meaning.  Finally,  it  helps  students  distinguish  between  given 
information  in  the  message  and  their  inferences  or  interpretations  of  the  message.  In  doing  so, 
(CT)2  encourages  a  deeper  processing  of  messages  than  people  often  engage  in.  The  finding  that 
memory  of  messages  is  enhanced  after  participating  in  (CT)2  is  a  strong  indicator  that  those 
messages  have  been  processed  at  a  deeper  level. 

(CT)"  also  seems  to  inhibit  the  production  of  inferences  that  go  well  beyond  what  is 
explicitly  given  in  the  message.  Participants  who  took  the  (CT)2  training  made  significantly 
fewer  unjustified  inferences  than  participants  assigned  to  the  other  two  training  conditions. 
Examinations  of  the  responses  reveals  that  (CT)2  participants  did  make  inferences;  however,  they 
justified  them  by  pointing  out  explicit  information  given  in  the  message  that  supported  their 
inferences.  Therefore,  (CT)2  appears  to  encourage  discrimination  of  what  is  “known”  or  “given” 
from  what  might  be  added  (i.e.,  inferred)  by  the  perceiver. 

The  ability  to  make  this  distinction  is  critical  to  avoiding  many  errors  of  reasoning.  In 
these  data,  for  example,  participants  who  were  inclined  to  make  unjustified  inferences  expressed 
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beliefs  that  were  clearly  erroneous.  The  fourth  message  of  the  assessment  instrument  is  an 
excerpt  of  President  Bush’s  introduction  to  the  National  Security  Strategy  that  was  written  post 
September  11,  2001,  but  before  the  war  in  Iraq.  When  asked  what  President  Bush  meant  by 
“poverty  does  not  make  poor  people  into  terrorists  and  murderers,”  several  participants  provided 
the  following  answers,  missing  the  President’s  point  entirely.  He  means,  “he  does  not  intend  to 
fight  poor  people,”  or  “he  does  not  intend  to  target  poorer  countries.”  The  meaning  of  the 
President’s  statement  is  clear  from  other  infonnation  given  in  the  text:  “Yet  poverty,  weak 
institutions,  and  corruption  can  make  weak  states  vulnerable  to  terrorist  networks. . If 
respondents  had  attended  closer  to  what  was  given  in  the  message,  and  knew  that  they  were 
making  an  inference  about  his  intentions  to  fight  the  poor,  they  would  have  realized  that  their 
inference  was  incorrect,  i.e.,  that  it  was  not  the  point  President  Bush  was  making.  In  other  words, 
these  results  suggest  that  (CT)"  encourages  evidence-based  reasoning. 

The  lack  of  significant  differences  among  the  groups  on  the  questions  of  the  belief  variable 
is  inconsistent  with  the  other  findings.  One  would  expect  those  who  better  remembered  specific 
information  and  stuck  to  the  message  closely  when  making  inferences  would  also  be  more 
sensitive  to  vague  or  otherwise  weak  information  given  in  a  message.  One  would  expect  those 
same  people  to  question  whether  they  should  believe  information  that  they  perceived  as  weak. 
However,  the  data  did  not  uphold  this  prediction.  It  is  possible  that  the  evaluation  questions  did 
not  adequately  elicit  judgments,  which  have  been  shown  to  encourage  questioning  in  previous 
studies  (See  the  validation  investigation  in  Volume  Two  of  this  series).  Alternatively,  it  may  be 
that  the  training  for  the  particular  critical  thinking  skill  that  was  the  focus  of  this  research  (Frame 
the  Message)  is  not  particularly  effective  at  eliciting  questions  about  the  validity  of  material. 

This  investigation  leaves  this  research  question  unresolved. 

In  summary,  the  present  research  provided  an  evaluation  of  (CT)~  prototype,  in  terms  of 
how  well  it  is  likely  to  be  accepted  by  military  users  and  its  effectiveness  at  improving  CT. 
Future  research  should  focus  on  comparing  the  effectiveness  of  other  CT  training  programs  to 
(CT)2  using  alternative  indicators  of  CT. 
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THE  NEXT  STEP: 

EXPANDED  WEB-BASED  SYSTEM  FOR  TRAINING  CRITICAL 

THINKING 


This  is  the  third  volume  of  a  three  volume  report  which  documents  the  foundational 
research  for  the  development  of  a  web-based  training  system  for  critical  thinking.  This  research 
(1)  reviewed  the  literature  on  critical  thinking,  (2)  produced  an  innovative,  testable,  and  partially 
validated  model  of  CT,  (3)  identified  eight  critical  thinking  skills  that  are  important  and 
problematic  to  Army  battle  command,  and  (4)  developed  and  evaluated  a  web-based  training 
prototype  that  targets  the  development  of  two  critical  thinking  skills  for  Army  battle  command, 
Frame  the  Message  and  Recognize  Gist  in  Material 

This  research  underlies  the  expanded  version  of  (CT)"  (Fischer,  et  ah,  2008)  which  offers 
training  for  all  eight  of  the  critical  thinking  skills.  In  addition  to  the  two  in  the  prototype  (CT)", 
the  following  skills  are  also  trained: 

1 .  Develop  an  explanation  that  ties  information  elements  together  in  a  plausible  way 

2.  Generalize  from  specific  instances  to  broader  classes 

3.  Visualize  plans  to  see  if  they  achieve  goals 

4.  Recognize  fallibility  and  bias  in  own  opinion 

5.  Adopt  multiple  perspectives 

6.  Decide  when  to  seek  infonnation  based  on  its  value  and  cost 

Based  on  feedback  from  the  prototype  evaluations,  extensive  changes  were  made  to  the 
design  of  the  training  system.  For  example,  more  exercises  within  each  module  were  developed 
to  provide  the  opportunity  for  practice,  performance  feedback,  and  reinforcement.  To  help 
sustain  user  interest,  each  module  is  now  a  multimedia  presentation  with  external  inputs  (voice, 
video,  animation,  text,  and  data).  The  ability  of  the  user  to  maneuver  around  the  system  was 
improved. 

In  addition,  a  user’s  manual  was  developed  (Fischer,  Spiker,  Harris,  &  McPeters,  2004). 
This  ensures  that  (CT)  can  be  easily  used  by  schoolhouse  instructors  and  commanders  who  may 
wish  to  use  it  to  train  their  units,  The  manual  includes  an  overview  of  the  training,  and  the 
learning  objectives  targeted  by  the  system.  It  also  guides  the  user  in  utilizing  the  training  to 
obtain  maximum  change  in  the  critical  thinking  of  the  targeted  students.  Expected  outcomes, 
level  of  effort,  and  procedures  are  discussed. 

Formative  evaluations  were  a  major  component  in  the  design  and  implementation  of  the 
expanded  version  of  (CT)".  Before  proceeding  with  Web  development,  training  content  was 
reviewed  by  members  of  a  panel  composed  of  individuals  who  represented  military  experts  and 
the  target  population  of  students.  This  panel  of  reviewers  was  recruited  from  the  population  of 
military  retirees  who  develop  curriculum  for  the  Command  and  General  Staff  College.  The 
reviewers  assessed  the  content  of  each  module  for  technical  accuracy,  relevance  to  tactical  battle 
command,  interest,  educational  value,  and  comprehensibility.  The  clarity  of  the  cognitive  and 
behavioral  objectives  was  scrutinized,  as  were  the  answer  keys  to  the  practical  exercises  and 
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tests.  In  keeping  with  the  build-test-rebuild  approach  taken  throughout  the  project,  the  reviews 
and  revisions  were  conducted  iteratively  in  two  cycles. 

Formative  usability  evaluations  were  also  an  integral  part  of  the  software  design  and 
implementation  and  four  were  conducted  for  the  expanded  version  of  (CT)  .  A  user-centered 
design  process  was  employed  to  construct  the  web-based  (CT)2  (Norman  &  Draper,  1986).  The 
two  foci  of  user-centered  design  are  usefulness  and  usability.  Because  the  identification  of  the 
eight  critical  thinking  skills  addresses  usefulness,  the  user-centered  design  process  for  this 
software  focused  on  usability.  Achieving  high  usability  with  complex  content  is  almost 
invariably  the  product  of  extensive  testing,  modification,  and  refinement.  The  purpose  of  this 
step  was  to  detennine  whether  our  design  and  implementation  leads  to  correct  use.  Are  users 
taking  full  advantage  of  the  designed  capabilities?  If  not,  why  not?  Do  users  correctly  understand 
the  content?  What  actions  are  available  to  them,  and  what  will  be  the  consequences  of  their 
actions? 

The  expanded  training  system  has  received  positive  evaluations  from  users  (see  Fischer,  et 
ah,  2008).  The  training  has  the  advantage  of  being  accessible  by  anyone  with  Internet  access, 
which  means  that  it  could  be  used  by  any  Army  or  reservist  field  unit  anywhere  in  the  world.  It 
could  also  be  used  as  distance  learning  or  for  residents  at  Anny  schoolhouses  such  as  the 
Command  and  General  Staff  College,  West  Point,  and  the  Army  War  College.  In  short,  we 
believe  it  has  wide  applicability.  In  addition,  the  critical  thinking  skills  it  trains  are  universally 
important  in  situation  assessment,  decision  making,  and  problem  solving,  whatever  the 
population.  (CT)2  could  be  easily  adapted  to  other  user  groups  by  tailoring  the  scenarios  and 
examples. 
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APPENDIX  A:  CRITICAL  THINKING  SKILLS  IN  THE  LITERATURE 


Interpretation  Skills 

1 .  Recognize  central  thesis  in  a  work 

2.  Break  goal  into  sub  goals 

3.  Strip  verbal  argument  of  irrelevancies  and  rephrase  it  in  essential  terms 

4.  Extract  meaning  from  context 

5.  Understand  contextual  nuances 

6.  Frame  the  message 

7.  Probe  question  or  problem  to  obtain  clarifying  information 

8.  Question  deeply 

9.  Redefine  problem  and  goal 

10.  Seek  clear  statement  of  the  question 

1 1 .  Understand  intended  definition  of  certain  words 

12.  Discern  when  a  term  is  used  with  different  meanings 

13.  Recognize  need  for  operational  definition 

14.  Identify  and  challenge  assumptions 

15.  Identify  unstated  assumption  in  a  discussion 

16.  Identify  missing  information  from  an  argument 

17.  Identify  premises  and  conclusions 

18.  Identify  missing  premises 

19.  Analyze  ambiguities  in  arguments 

20.  Critique  to  distinguish  reliable  from  unreliable  assumptions 

21.  Distinguish  fact/opinion/assumption  elements  in  an  argument 

22.  Distinguish  relevant  from  irrelevant  information 

23.  Examine  evidence  to  distinguish  anecdote  from  fact 

24.  Determine  whether  a  statement  is  overly  vague  or  overly  specific 

25.  Identify  own  assumptions  and  biases 

26.  Identify  emotional  language  in  external  sources 

27.  Distinguish  between  validity  of  a  belief  and  intensity  with  which  it  is  held 

28.  Seek  disconfirming  evidence 

29.  Know  when  causal  claims  can  and  can’t  be  made 
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Reasoning  Skills 

1 .  Understand  limits  of  extrapolation 

2.  Reason  by  finding  analogous  arguments  to  bolster  conclusion 

3.  Refine  generalizations  and  avoid  oversimplification 

4.  Apply  general  principles  to  specific  cases 

5.  Generalize  from  specific  instances  to  broader  classes 

6.  Determine  whether  a  simple  generalization  is  warranted 

7.  Draw  inductive  inference  from  observations 

8.  Reason  by  taking  representative  samples 

9.  Distinguish  between  deductive  and  inductive  reasoning 

10.  Reason  by  deductive  logic  to  draw  conclusions  from  premises 

1 1 .  Reason  dialogically  to  identify  and  compare  perspectives 

12.  Reason  dialectically  to  evaluate  points  of  view 

13.  Trace  logic  in  an  argument 

14.  Determine  whether  a  statement  follows  from  premises 

15.  Distinguish  between  logically  valid  and  invalid  inferences 

16.  Check  consistency  of  infonnation  in  the  problem 

17.  Avoid  ad  hominem  reasoning  fallacy  (consider  argument  not  the  person) 

18.  Avoid  false  dichotomy  reasoning  fallacy  (artificially  reduce  the  number  of  choices) 

19.  Avoid  guilt  by  association  reasoning  fallacy 

20.  Avoid  emotional  appeal  reasoning  fallacy 

2 1 .  Identify  instances  of  faulty  thinking 

22.  Mentally  simulate  plans  to  see  if  they  achieve  goals 

23.  Mentally  generate  a  structure  of  possibilities  that  presently  don’t  exist 

24.  Mentally  simulate  probable  consequences  of  alternative 

25.  Develop  and  use  mental  models 

26.  Recognize  bias  in  hindsight  analysis 

27.  Reason  from  starting  point  with  which  one  disagrees 

28.  Recognize  fallibility  of  own  opinion 

29.  Recognize  probability  of  bias  in  own  opinion 

30.  Recognize  transitive  relationships 

3 1 .  Inference  to  “best  explanation”  that  explains  facts  better  than  all  other  alternatives 

32.  Make  reasoned  value  judgment  by  considering  background,  consequences,  principles 

33.  Understand  difference  between  reasoning  and  rationalizing 

34.  Analyze  problem  by  working  backward 

35.  Determine  when  evidence  is  insufficient  to  warrant  sound  conclusion 
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Assessment  Skills 

1 .  Know  value  and  cost  of  information  and  how  and  when  to  seek  it 

2.  Know  when  new  information  supports/refutes  conclusion 

3.  Consider  new  evidence  as  it  becomes  available 

4.  Weigh  multiple  factors  when  necessary 

5.  Perform  means-ends  analysis  to  check  status 

6.  Support  general  assertions  with  details 

7.  Frame  decision  in  alternative  ways 

8.  Assess  an  assertion’s  truthfulness  based  on  accuracy  of  relevant  facts 

9.  Assess  an  assertion’s  truthfulness  based  on  its  degree  of  precision 

10.  Assess  an  assertion’s  truthfulness  based  on  presence  of  unbiased  evidence 

11.  Assess  an  assertion’s  truthfulness  based  on  having  credible  sources 

12.  Assess  an  assertion’s  truthfulness  based  on  its  logical  consistency 

13.  Assess  an  observation’s  credibility  based  on  short  time  between  observation  and  report 

14.  Assess  an  observation’s  credibility  based  on  first-hand  report  by  observer 

15.  Assess  an  observation’s  credibility  based  on  minimal  interference 

16.  Assess  an  observation’s  credibility  based  on  reporter’s  belief  that  observation  was 
accurate 

17.  Assess  an  observation’s  credibility  based  on  corroboration  by  other  sources 

18.  Assess  credibility  of  information  source  based  on  author’s  reputation  for  accuracy 

19.  Assess  credibility  of  information  source  based  on  being  in  author’s  field  of  expertise 

20.  Assess  credibility  of  information  source  based  on  absence  of  conflict  of  interest 

21.  Assess  credibility  of  information  source  based  on  known  risk  to  author’s  reputation 

22.  Assess  credibility  of  information  source  based  on  data-gathering  and  processing  methods 

23.  Assess  credibility  of  information  source  based  on  agreement  with  other  sources 

24.  Assess  strength  of  conclusion  based  on  reasonableness  of  assumptions 

25.  Assess  strength  of  conclusion  based  on  consistency  with  known  facts 

26.  Assess  strength  of  conclusion  based  on  alternatives  are  inconsistent  with  known  facts 

27.  Assess  strength  of  conclusion  based  on  its  ability  to  explain  the  evidence 

28.  Assess  strength  of  argument  based  on  clarity  of  meaning 

29.  Assess  strength  of  argument  based  on  identity  of  stated  and  unstated  conclusions 

30.  Assess  strength  of  argument  based  on  identity  of  premises  supporting  conclusions 

3 1 .  Assess  strength  of  argument  based  on  identity  of  unstated  assumptions 

32.  Assess  strength  of  argument  based  on  reliability  and  reasonableness  of  inferences 

33.  Assess  strength  of  argument  based  on  other  relevant  arguments 

34.  Assess  quality  of  judgment  based  on  kind  of  judgment  being  made 

35.  Assess  quality  of  judgment  based  on  presence  of  indicators  related  to  the  criteria 

36.  Assess  quality  of  judgment  based  on  pattern  among  indicators 

37.  Assess  quality  of  judgment  based  on  degree  of  match  with  criteria 
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Meta-Cognitive  Monitoring  Skills 

1 .  Look  beyond  first  obvious  explanation  to  consider  alternative  interpretations 

2.  Identify  the  need  to  think  hard 

3.  Develop  perspective  to  explore  the  implications  of  beliefs,  arguments,  or  theories 

4.  Ask  questions  and  be  willing  to  ponder  (e.g.,  use  scientific  method) 

5.  Construct  story  that  ties  all  infonnation  elements  together  in  plausible  way 

6.  Generate  summaries 

7.  Generate  alternative  explanations 

8.  Generate  multiple  ideas 

9.  Adopt  multiple  perspectives 

10.  Consider  multiple  sides  of  an  issue 

1 1 .  Stay  relevant  to  the  main  point 

12.  Take  total  situation  into  account 

13.  Monitor  events  for  consistency  with  expectations 

14.  Monitor  own  understanding  of  problem 

15.  Compare  analogous  situations  and  transfer  pertinent  learning  to  new  contexts 

16.  Know  that  observations  are  more  credible  than  inferences  based  on  them 

17.  Read  between  the  lines 

18.  Determine  whether  argument  depends  on  an  ambiguity 

19.  Understand  differences  among  conclusions,  assumptions,  and  hypotheses 
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APPENDIX  B: 

TEACHING  METHODS  FOR  PROMOTING  CRITICAL  THINKING 


Teaching  Method 

Critical  Thinking  Skills 

Reference 

Type  of 
Evidence 

Guided  Web  Exploration:  A  basic  set  of  navigation 
buttons  to  structure  web  exploration  used  in  the  context 
of  hypothetical  tasks  on  the  Internet. 

1 .  Determining  the  relevant  from  the  irrelevant 

2.  Decide  when  to  seek  info,  based  on  its  value  and  cost 

3.  Decide  when  to  consider  new  information  or 
evidence 

4.  Seek  disconfirming  evidence 

Baker  (1997) 

Testimonial 

Conference  style  learning/Seminar 

1.  Make  a  reasoned  value  judgment  by  considering 
background,  consequences,  principles 

Underwood  & 

Wald  (1995) 

Testimonial 

Evaluating  Arguments —  Students  are  required  to 
analyze  and  evaluate  competing  arguments  on  an  issue. 
Class  discussions  about  the  arguments  are  then  engaged. 
Then,  students  write  a  paper  in  which  they  defend  one  of 
the  arguments. 

1 .  Assess  the  strength  of  a  conclusion  or  argument 

Bernstein  (1995) 

Testimonial 

Evaluating  arguments  as  problem  solving  methods. 
Students  are  trained  to  see  competing  arguments  as 
defenses  of  solutions  that  come  out  of  a  particular 
representation  of  a  problem. 

2.  Frame  decision  in  alternative  ways 

3.  Generate  alternative  explanations  or  interpretations 

Bernstein  (1995) 

Testimonial 

Negotiation  Skill.  Students  learn  about  various  sides  of 
an  issue  such  as  animal  research  ethics.  They  see  videos 
and  read  and  evaluate  arguments.  They  then  form 
simulated  animal  research  and  care  committees,  taking 
on  a  role  opposite  to  their  personal  position.  They  first 
have  to  draft  a  set  of  guidelines  within  the  4-person 
committee.  They  then  have  to  decide  on  four  protocols 
submitted  to  the  committee.  They  are  reminded  to  use 
“Getting  to  Yes”  skill  in  identifying  common  interests. 

The  model  insists  on  treating  all  parties  with  respect  and 
being  genuinely  empathic  with  their  interests. 

1.  Assess  the  strength  of  a  conclusion  or  argument 

2.  Adopt  multiple  perspectives 

3.  Reason  from  starting  point  with  which  one  disagrees 

Bernstein  (1995) 

Testimonial 
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Teaching  Method 

Critical  Thinking  Skill 

Reference 

Type  of 
Evidence 

Debate  —  the  National  Debating  Team  method, 
emphasizes  a  dialectical  approach  to  arriving  at  truth. 

This  approach  emphasizes  and  judges  arguments  based 
on  evidence  brought  forward  to  arrive  at  conclusions. 

1 .  Base  reasoning  on  observations  because  they  are 
more  credible  than  inferences  based  on  them 

2.  Judge  when  evidence  is  insufficient  to  warrant 
conclusion 

Colbert  (1986) 

Empirical. 
Positive  Effect 
on  WGCTA 
compared  to 
Aristotelian 
approach 

Debate  —  Aristotle’s  rhetoric  approach  to  debate  which 
acknowledges  evidence  but  also  emphasizes  the  use  of 
persuasive  techniques  in  argument. 

1.  Use  persuasive  techniques  in  argument 

Colbert  (1986) 

Empirical. 
Positive  Effect 
on  WGCTA 
compared  to 
control  group 

Teaching  argument  concepts  like  claim  and  evidence 
using  modeling  of  argument  evaluation,  feedback  of 
students  argument  evaluations  about  claim  and  evidence, 
and  exercise  in  which  claims  and  evidence  are  identified 

1 .  Identify  assumptions  and  challenge  validity 

2.  Distinguish  among  conclusions,  assumptions, 
hypotheses 

3.  Distinguish  between  facts  and  opinions 

4.  Distinguish  relevant  from  irrelevant  information 

5.  Judge  when  causal  claims  can  be  made 

6.  Judge  when  evidence  is  insufficient 

7.  Base  reasoning  on  observations 

Klein,  Olson,  & 
Stanovich  (1997) 

Empirical. 
Increased 
ability  to 
evaluate 
arguments 

Teaching  strategies  for  planning  and  monitoring 
evaluation  of  an  argument  such  as  “I  will  be  a  careful 
idea  shopper,”  “I’ll  try  to  figure  out  what  the  author 
wants  me  to  believe.” 

1 .  Identify  assumptions  and  challenge  validity 

2.  Distinguish  among  conclusions,  assumptions, 
hypotheses 

3.  Distinguish  between  facts  and  opinions 

4.  Distinguish  relevant  from  irrelevant  information 

5.  Judge  when  causal  claims  can  be  made 

6.  Judge  when  evidence  is  insufficient 

7.  Base  reasoning  on  observations 

Klein,  Olson,  & 
Stanovich  (1997) 

Empirical. 
Increased 
ability  to 
evaluate 
arguments,  and 
increased 
creation  of 
evidence-based 
arguments 

Within  a  course,  encouraging  doubt  of  written  material, 
encouraging  exploration  of  own  opinions,  encourage 
question  and  challenge  of  the  instructor 

1 .  Identify  assumptions  and  challenge  their  validity 

2.  Identify  own  assumptions  and  biases 

3.  Adopt  multiple  perspectives 

4.  Identify  common  fallacies 

Huff  (2000) 

Empirical. 

Improved 

CCTST  scores 
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Teaching  Method 

Critical  Thinking  Skill 

Reference 

Type  of 
Evidence 

Collaborative  learning—  students  were  instructed  to 
discuss  why  they  thought  as  they  did  regarding  solutions 
to  the  worksheet  problems.  They  were  also  instructed  to 
listen  carefully  to  each  member  of  the  group  and  to 
reconsider  their  own  judgments.  Every  group  member 
was  required  to  contribute  his  or  her  ideas. 

1 .  Recognize  fallibility  and  likely  bias  of  own 
opinion 

2.  Assess  quality  of  own  judgment  based  on 
indicators  related  to  criteria 

Gokhale  (1995) 

Empirical. 
Produced  better 
scores  on  CT 
items  than 
individual 
training 

Students  are  asked  to  explain  why,  identify  causes, 
determine  the  nature  of,  identify  why  something  is 
important,  identify  the  implications  of,  determine  how 
something  is  affected,  what  is  the  solution? 

1 .  Seek  clear  statement  of  question 

2.  Construct  story  that  ties  all  information 

3.  Judge  when  causal  claims  can  be  made 

4.  Use  deductive  logic 

5.  Draw  inductive  inferences 

Hittner  (1999) 

Empirical.  Self 
report  of 
students  taking 
the  course 
indicated 
improvement  in 
CT. 

Structure  curriculum  around  ideas,  not  facts  or  themes. 
Model  CT.  Use  demonstrations  as  opportunities  for 
mutual  thought.  Discussion  groups 

No  CT  skills  were  trained,  but  the  predisposing 
attitudes  of  curiosity,  skepticism,  reflective  style, 
confidence,  and  willingness  to  engage  in  neutral  effort 
were. 

Gray  (1993) 

Testimonial 

Introduce  controversy  in  class,  ask  provocative 
questions,  ask  questions  that  inform  what  the  student  is 
learning,  use  essays  not  multiple  choice,  do  this  in  all 
classes 

No  CT  skills  were  trained,  but  factors  that  "encourage 
CT"  include:  introducing  controversy,  asking 
provocative  questions,  using  testing  instruments  which 
promote  active  learning  (essay),  and  integrating  efforts 
across  the  curriculum. 

Browne,  Hoag,  & 
Berilla  (1995) 

Testimonial 
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Teaching  Method 

Critical  Thinking  Skill 

Reference 

Type  of 
Evidence 

Provide  explicit  instruction  in  forming  generalizations. 
Students  given  lecture  and  text,  guided  practice  in 
groups,  criteria  checklists  for  generalizing,  discussion 
groups. 

1 .  Refine  sweeping  generalizations 

2.  Generalize  from  specific  instances  to  broader 
classes 

3.  Determine  whether  a  simple  generalization  is 
warranted 

4.  Distinguish  between  facts  and  opinions 

5.  Distinguish  among  conclusions,  assumptions,  and 
hypotheses 

6.  Distinguish  between  validity  of  belief  and  intensity 
with  which  it  is  held 

7.  Draw  inductive  inference  from  observations 

8.  Distinguish  between  logically  valid  and  invalid 
inferences 

9.  Base  reasoning  on  observations 

10.  Detect  inconsistencies 

11.  Judge  when  evidence  is  insufficient 

12.  Judge  when  causal  claims  and  can’t  be  made 

Kemp  &  Sadoski 
(1991) 

Empirical. 
Improved 
generalization, 
but  overall  CT 
did  not  improve 
as  measured  by 
the  CCTT. 

Infusion  approach.  Instruction  included  special  thinking 
activities  that  involved  problem  solving,  analysis  of 
experiments,  and  data  handling.  Different  contexts  were 
used 

1 .  Determine  whether  a  simple  generalization  is 
warranted 

2.  Judge  when  evidence  is  insufficient 

3.  Distinguish  between  logically  valid  and  invalid 
inferences 

4.  Reason  by  taking  representative  samples 

5.  Base  reasoning  on  observations 

6.  Distinguish  between  inductive  and  deductive 
reasoning 

7.  Use  deductive  logic  to  draw  conclusions 

8.  Distinguish  among  conclusions  assumptions  and 
hypotheses 

9.  Distinguish  between  facts  and  opinions 

10.  Identify  assumptions  and  challenge  their  validity 

1 1 .  Identify  and  assess  validity  of  unstated 
assumptions 

12.  Detect  missing  operational  definitions 

13.  Identify  missing  information 

14.  Distinguish  relevant  from  irrelevant 

Zohar,  Weinberger, 
&  Tamir  (1994) 

Empirical. 
Improvement 
on  general  and 
domain  specific 
tests  of  CT. 
Transfer  was 
also  achieved. 
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Teaching  Method 

Critical  Thinking  Skill 

Reference 

Type  of 
Evidence 

Generate  thought  provoking  questions.  Stems  of 
questions  include:  compare  and  contrast,  create 
analogies,  judge  importance,  provide  evidence,  generate 
multiple  perspectives,  predict  outcomes,  etc. 

1 .  Redefine  problem 

2.  Break  overall  goal  into  subgoals 

3.  Seek  clear  statement 

4.  Reason  dialogically 

5.  Reason  dialectically 

6.  Distinguish  between  inductive  and  deductive 

7.  Distinguish  between  valid  and  invalid 

8.  Use  analogous  arguments 

9.  Explore  implications  of  beliefs 

10.  Generate  alternative  explanations 

1 1 .  Identify  assumptions  and  challenge 

12.  Identify  own  assumptions 

13.  Judge  when  causal  claims  can  be  made 

14.  Draw  inductive  inferences 

15.  Adopt  multiple  perspectives 

16.  Support  general  assertions  with  facts 

17.  Assess  strength  of  conclusion 

King  (1995) 

Testimonial 

Assign  roles  in  a  group  that  involve  different  thinking 
tasks.  These  include  task  definer,  strategist,  monitor, 
challenger 

1 .  Seek  a  clear  statement  of  the  problem 

2.  Distinguish  relevant  from  irrelevant 

3.  Identify  assumptions  and  challenge 

4.  Identify  missing  information 

5.  Detect  inconsistencies 

6.  Use  deductive  logic 

7.  Reason  dialogically 

8.  Draw  inductive  inference 

9.  Monitor  own  understanding 

10.  Support  general  assertions  with  facts 

Riesenmy, 

Mitchell,  Hudgins, 

&  Ebel  (1991) 

Empirical 

Methodological  education  in  a  probabilistic  science  — 
graduate  education 

No  CT  skills  were  trained.  The  training  examined  was 
that  provided  in  four  graduate  programs:  law, 
medicine,  psychology,  and  chemistry. 

Lehman,  Lempert, 

&  Nisbett  (1988) 

Empirical. 

Education  improves 
reasoning  ability 

Group  discussion  vs.  traditional  lecture  method 

No  CT  skills  were  trained.  Treatment  consisted  of 
group  discussion  and  group  writing. 

Garside  (1996) 

Empirical. 

No  improvement  in 
CT  ability 

Critical  Analysis — Have  students  critique  material, 
focusing  on  questions  that  require  deep  thinking,  e.g., 
compare  and  contrast,  why,  what  is  most  pressing  need, 
etc. 

Not  stated.  Presumably: 

1 .  Mentally  generate  structure  of  possibilities. 

2.  Mentally  simulate  probable  consequences  of  an 
alternative. 

Jones  &  Safrit 
(1994) 

Testimonial 
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Teaching  Method 

Critical  Thinking  Skill 

Reference 

Type  of 
Evidence 

Debate  Teams — Have  students  debate  topic,  particularly 
effective  if  learners  are  asked  to  serve  on  a  debate  team 
whose  position  differs  from  that  of  their  own. 

Not  stated.  Presumably: 

1 .  dialogical  reasoning 

Jones  &  Safrit 
(1994) 

Testimonial 

Dramatization — Students  view  a  portion  of  a  video  and 
then  script  and  act  out  a  potential  ending 

Not  stated,  unclear. 

Jones  &  Safrit 
(1994) 

Testimonial 

Action  Maze — Students  are  divided  into  small  groups 
and  presented  with  situations.  They  must  identify  their 
responses  to  the  situation  as  well  as  the  consequences  of 
each  alternative. 

1 .  Mentally  generate  structure  of  possibilities. 

2.  Mentally  simulate  probable  consequences  of  an 
alternative. 

Jones  &  Safrit 
(1994) 

Testimonial 

Critical  Incident — A  portion  of  a  situation  is  presented 
to  learners  who  then  must  make  a  decision  about  solving 
the  problem.  The  teacher  then  points  out  additional 
information  that  the  learner  may  have  overlooked. 

Not  stated.  Presumably: 

1 .  Identify  missing  information. 

2.  Support  general  assertions  with  specific  facts. 

3.  Seek  disconfirming  evidence. 

Jones  &  Safrit 
(1994) 

Testimonial 

Scenario  Building — Develop  a  detailed  written 
description  of  a  specified  situation 

Not  stated,  unclear. 

Jones  &  Safrit 
(1994) 

Testimonial 

Socratic  Questioning — Presenting  questions  that  evoke 
ideas  and  viewpoints,  explore  applications,  and  provide 
implications. 

No  CT  skills  trained. 

Jones  &  Safrit 
(1994) 

Testimonial 

Creative  Visualization — Creating  mental  images  of 
hypothetical  situations 

1 .  Mentally  generate  a  structure  of  possibilities. 

Jones  &  Safrit 
(1994) 

Testimonial 

Journal  Writing — reflect  on  actions  and  behaviors 
related  to  program  topic 

No  CT  skills  trained. 

Jones  &  Safrit 
(1994) 

Testimonial 

Quotations  or  cartoons — presentation  of  thought 
provoking  quotations  or  cartoons  with  discussion 
following. 

No  CT  skills  trained. 

Jones  &  Safrit 
(1994) 

Testimonial 

Inventing — Student  invents  or  designs  new  products, 
services,  or  teaching  strategies. 

1.  Mentally  generate  structure  of  possibilities. 

2.  Mentally  simulate  plans  to  see  if  they  achieve  goals. 

3.  Support  general  assertions  with  specific  facts. 

Jones  &  Safrit 
(1994) 

Testimonial 

Pluses,  minuses,  and  alternatives — List  the  pluses, 
minuses,  and  implications  (PMI)  of  each  alternative 
decision. 

No  CT  skills  trained. 

Jones  &  Safrit 
(1994) 

Testimonial 

Train  abstract  logic  principles  coupled  with  examples  of 
selection  problems 

Wason  card  test — deductive  logic 

Cheng,  Holyoak, 
Nisbett,  &  Oliver 
(1986) 

Empirical. 

Improves 

deductive 

reasoning 
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Teaching  Method 

Critical  Thinking  Skill 

Reference 

Type  of 
Evidence 

Competitive  forensics  in  communication  classes 

argument  creation  and  evaluation 

Allen,  Berkowitz, 
Hunt,  &  Louden 
(1999) 

Empirical. 
Improves  CT  on 
Watson  Glaser 
more  than 
argumentation 
and  public 
speaking 

Argumentation  training  in  communication  classes 

argument  creation  and  evaluation 

Allen,  Berkowitz, 
Hunt,  &  Louden 
(1999) 

Empirical, 
improves  CT  on 
Watson  Glaser 
more  than 
public  speaking 

Public  speaking  in  communication  classes 

argument  creation  and  evaluation 

Allen,  Berkowitz, 
Hunt,  &  Louden 
(1999) 

Empirical, 
improves  CT 

Training  pragmatic  reasoning  schemas  like  a  permission 
schema  and  an  obligation  schema 

deductive  reasoning  with  the  conditional,  standard 
logic  skills 

Cheng,  Holyoak, 
Nisbett,  &  Oliver 
(1986) 

Empirical,  but 
training  in 
abstract 
principles  is 
useful  only  if  it 
is  paired  with 
training  on 
pragmatic 
reasoning 
schemas 
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APPENDIX  C: 

IDENTIFICATION  OF  EIGHT  HIGH-PAYOFF  CRITICAL 
THINKING  SKILLS  FOR  BATTLE  COMMAND 

(Taken  from  Fischer,  Spiker  and  Riedel  (2008b)  and  duplicated  here  for  convenience.) 

A  two-tiered  approach  was  used  to  select  a  set  of  high-payoff  critical  thinking  skills  used 
in  battle  command.  At  the  first  tier,  the  field  was  narrowed  by  identifying  six  broad  classes  of 
tasks  from  the  original  13  that  were  evaluated  by  the  participants  in  the  Fort  Hood  interviews. 
The  first  cut  was  made  by  jointly  considering  two  criteria  based  on  the  Fort  Hood  survey  data. 
First,  the  mean  importance  ratings  of  the  13  broad  classes  of  skills  were  taken  into  account. 
Second,  we  considered  the  number  of  officers  who  reported  that  they  had  observed  problems  in 
executing  the  13  broad  classes  of  tasks. 

The  mean  importance  ratings  are  presented  in  the  far  right-hand  column  of  Table  4  (of 
Volume  Two)  and  their  significance  was  previously  discussed.  The  number  of  officers  who 
reported  they  had  observed  problems  with  each  task  class  is  also  given  in  that  table.  Figure  C-l 
plots  each  task  class  in  terms  of  its  rated  importance  (x-axis)  and  number  of  reported  problems 
(y-axis).  To  systematize  the  selection  process  and  equally  weight  the  two  criteria,  cross-hairs  are 
placed  on  the  figure  to  demark  lines  along  the  x-  and  y-axes  where  reasonable  cutpoints  could  be 
placed.  Using  this  approach,  the  broad  classes  of  skills  located  in  the  upper  left  quadrant  were 
selected.  As  shown  in  Figure  C-l,  there  are  six  such  classes  that  satisfy  the  criteria  of  having  a 
sufficiently  low  (where  1  =  high  importance,  5  =  low  importance)  importance  rating  (below  1 .7 
indicating  officers  thought  they  were  very  important)  and  a  large  incidence  of  problems  (more 
than  7  officers  out  of  18  reported  they  had  observed  problems).  Using  these  criteria,  the  six 
classes  of  critical  thinking  skills  that  deserve  the  most  vigorous  investigation  in  future  research 
are: 


•  Framing  the  problem 

•  Inductive  reasoning 

•  Mental  simulation 

•  Avoiding  reasoning  fallacies 

•  Meta-reasoning 

•  Extracting  meaning. 


Figure  C-l.  Ratings  of  importance  (x)  by  reported  problems  (y)  for  13  broad  classes  of 
critical  thinking  skills. 

However,  over  40  tasks  may  be  associated  with  these  six  broad  classes.  Thus,  a  second  cut 
was  applied  to  narrow  the  field  to  a  set  of  particular  skills  on  which  to  focus  further  research.  We 
relied  heavily  upon  the  officer’s  incident  reports  and  comments  about  each  class  of  skill  to  make 
the  final  selection  of  eight  critical  thinking  skills  that  would  ultimately  be  the  focus  of  future 
research.  In  addition  to  participant’s  comments,  The  following  four  criteria  were  applied.  The 
greatest  weight  was  given  to  the  second  selection  criteria.  Hence,  each  of  the  eight  tasks  was 
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primarily  chosen  because  it  was  used  in  a  particular  battle  command  task  for  which  deficiencies 
had  been  observed  by  the  interviewees. 

•  all  six  classes  identified  should  be  represented  in  the  final  selection 

•  the  task  should  be  used  in  the  performance  of  particular  battle  command  tasks  identified 

as  problematic  in  the  Fort  Hood  survey 

•  the  underlying  cognitive  processes  of  the  task  should  be  potentially  measurable 

•  the  task  should  be  applicable  to  future  research  and  training  development  efforts. 

Application  of  these  criteria  resulted  in  the  identification  of  eight  high-payoff  critical 
thinking  skills,  listed  in  the  middle  column  of  Table  C-l.  The  six  broad  classes  can  be  seen  in  the 
left-hand  column,  with  associated  battle  command  tasks  noted  in  the  right-hand  column.  These 
tasks  come  from  a  taxonomy  of  36  battle  command  tasks  that  were  consistently  found  to  have 
serious  human  perfonnance  problems  during  post  and  field  exercises  (Fallesen,  1993).  The  task 
list  was  organized  around  the  taxonomic  categories  of  mission,  terrain,  enemy,  own  forces, 
courses  of  action  (COAs),  battlefield,  and  planning  (Fischer,  Spiker,  &  Riedel,  2007b). 


Table  C-l.  Core  Critical  Thinking  Skills  Selected  for  Future  Research 


Critical  Thinking 
Class 

Selected  Critical 
Thinking  Skill 

Associated  Battle  Command  Tasks 

Frame  the 
Problem/Extract 

Meaning  from  Material 

Frame  the  Message 

•  Clarify  intent  of  commanders  1  and  2  levels 
up 

•  Review  a  mission  statement 

•  Determine  own  unit’s  area  of  responsibility 
from  OPORD 

•  Determine  constraints/restraints  placed  on 
mission  by  higher  HQ  from  OPORD 

Extract  Meaning  from 
Material 

Recognize  Gist  In  Material 

•  Review  a  mission  statement 

•  Clarify  commanders’  intent  1  and  2  levels  up 

Induction 

Develop  An  Explanation 

That  Ties  Information 
Elements  Together  In  A 
Plausible  Way 

•  Interpret  reports  of  recent  enemy  significant 
activities  in  area  of  interest 

•  Clarify  intent  of  commanders  one  and  two 
levels  up 

•  Read  the  battlefield 

•  Track  the  battlefield 

•  Interpret  reports  of  enemy  disposition 

Induction 

Generalize  From  Specific 
Instances  To  Broader 

Classes 

•  Interpret  reports  of  enemy  disposition 

•  Interpret  reports  of  recent  enemy  significant 
activities  in  area  of  interest 

Frame  the  Problem/ 
Simulation 

Visualize  to  Evaluate  Plans 

•  War  game  a  COA 

•  Develop  branches/sequences  for  each  COA 

•  Determine  ease  of  movement 

•  Perform  terrain  analysis  to  gain  an  appropriate 
perspective  that  supports  battle 

•  Predict  number  of  vehicles  that  can  fit  in  an 
engagement  area 
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Avoid  Reasoning 
Fallacies 

Challenge  One’s  Bias 

•  Change  own-unit  plans  based  on  new  tactical 
input 

•  Develop  COA’s 

•  Read  the  battlefield 

•  Track  the  battlefield 

•  Assess  the  situation 

•  Infer  status  of  enemy  forces 

Meta-Reasoning 

Examine  Other  Peoples’ 
Perspectives 

•  Interpret  reports  of  recent  enemy  significant 
activities  in  area  of  interest 

•  Infer  status  of  enemy  forces 

•  Interpret  reports  of  enemy  disposition 

•  Read  the  battlefield 

•  Track  the  battlefield 

•  Assess  the  situation 

Meta-Reasoning 

Decide  When  To  Seek 
Information  Based  On  Its 
Value  And  Cost 

•  Assess  the  situation 

•  Interpret  reports  of  enemy  disposition 

•  Interpret  reports  of  recent  enemy  significant 
activities  in  area  of  interest 

•  Read  the  battlefield 

•  Track  the  battlefield 

•  Infer  status  of  enemy  forces 

The  interview  data  obtained  at  Fort  Hood  confirmed  that  these  battle  command  tasks  are 
problematic,  i.e.,  officers  reported  that  they  had  observed  deficiencies  related  to  CT  in  these 
battle-command  tasks.  Interview  respondents  identified  the  battle  command  tasks  for  which 
deficiencies  in  CT  were  most  evident  and  troublesome.  As  noted  above,  these  reports  were 
central  to  identification  and  selection  of  the  eight  critical  thinking  skills.  Greater  weight  was 
given  to  battle  command  tasks  that  were  noted  by  several  officers.  For  example,  quite  a  few  of 
the  interviewees  reported  that  commander’s  intent  statements  were  difficult  to  write  and  to 
understand.  The  purpose  of  commander’s  intent  statements  is  to  provide  the  central  objective  and 
intention  of  the  mission.  However,  that  goal  is  rarely  achieved,  according  to  our  respondents. 
Another  commonly  noted  problematic  battle  command  task  was  the  inability  to  recognize  one’s 
own  biases,  which  manifested  itself  in  “fighting  the  plan.”  Many  interviewees  conveyed 
experiences  in  which  new  and  contrary  evidence  (e.g.,  changes  in  expected  enemy  avenue  of 
approach)  was  ignored  or  interpreted  to  fit  the  plan.  Thus,  decisions  were  made  to  continue  with 
the  original  battle  plan,  often  with  disastrous  results.  Table  C-2  further  defines  and  clarifies  the 
problems  associated  with  each  of  the  eight  selected  critical  thinking  skills.  A  definition  of  each 
task  is  given  in  the  third  column.  The  primary  battle  command  task  for  which  deficiencies  were 
noted  in  the  interview,  and  for  which  the  CT  task  was  chosen,  is  given  in  the  fourth  column.  The 
fifth  column  describes  the  kind  of  errors  and  deficiencies  that  were  reported  by  the  Fort  Hood 
officers. 
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Table  C-2.  Relationship  of  Battle  Command  Tasks,  CT  Issues  and  Selected  Critical 
Thinking  Skills 


Critical 

Thinking 

Skill 

(CTS) 

Critical 

Thinking 

Skill  Title 

Critical  Thinking  Skill 
Definition 

Primary  Battle 
Command  Task 

Battle  Command  Issue 

1 

Frame  The 
Message 

The  ability  to  identify  the 
essential  elements  of  a 
message,  understand  their 
relationships,  and  describe  a 
high  fidelity  representation 
of  the  message. 

Clarify  the  intent  of 
the  commanders  1 
and  2  levels  up 

1.  Difficulty  in  establishing 
clear  and  accurate  understanding 
of  CDR  intent. 

2.  Difficulty  in  conveying  clear 
CDR  intent. 

2 

Recognize 

Gist  In 

Material 

The  ability  to  sort  through 
the  details  in  a  message 
(written,  graphical,  visual, 
auditory,  and/or  tabular)  and 
extract  the  gist  therein. 

Restate  mission 
objectives  provided 
by  upper  echelon  to 
write  own  mission 
statement 

1.  Too  much  detail  in  OPORDS 
that  must  be  filtered  to  establish 
gist  that  supports  writing  of  own 
mission  statement 

2.  Too  little  time  at  lower 
echelons  to  accurately  extract 
essence  of  mission 

3 

Develop  An 

Explanation 

That  Ties 

Information 

Elements 

Together  In  A 

Plausible 

Way 

The  ability  to: 

•  Arrange  evidence 
logically 

•  Highlight  the  gaps  in 
knowledge. 

•  Develop  an  explanation 
or  multiple  explanations 
based  on  evidence 

Evaluate  explanation(s)  for 
plausibility 

The  ability  to: 

•  Arrange 
evidence  logically 

•  Highlight  the 
gaps  in 
knowledge. 

•  Develop 
explanations  based 
on  evidence 

•  Evaluate 
explanation(s)  for 
plausibility 

The  ability  to: 

•  Arrange  evidence  logically 

•  Highlight  the  gaps  in 
knowledge. 

•  Develop  an  explanation  or 
multiple  explanations  based  on 
evidence 

Evaluate  explanation(s)  for 
plausibility 

4 

Generalize 

From 

Specific 
Instances  to 
Broader 

Classes 

The  ability  to  recognize  and 
then  classify  specific 
facts/incidents/events  as  part 
of  a  general  category. 

Interpret  reports  of 
enemy  disposition 

1.  Fail  to  accurately  induce 
patterns  of  overall  movement 
based  on  report  instances 

2.  Tendency  to  disregard 
reports  that  do  not  match 
expectations 

3.  Tendency  to  inflate 
information  in  reports 

5 

Visualize  To 
Evaluate 

Plans 

The  ability  to  accurately 
create  mental  images  in  one’s 
mind  how  resources  will  be 
applied  and  events  will 
unfold  within  a  situation. 

Develop  scheme  of 
maneuver 

War  game  CO  As 

1.  Fail  to  visualize  events 

2.  Fail  to  include  sufficient 
detail  in  COAs 

3.  Fail  to  consider 
contingencies 

4.  Fail  to  consider  how  plans 
could  go  wrong 

5.  Generate  only  one  CO  A 

6.  Fail  to  consider  combat 
multipliers 

7.  Difficulty  in  keeping  track  of 
mobile  forces 
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6 

Challenge 
One’s  Bias 

Ability  to  reevaluate  one’s 
current  view  of  the  situation 
for  prejudice  or  bias  as  new 
information  is  received. 

Change  own-unit 
plans  based  on  new 
tactical  input 

1.  Tendency  to  “fight  the  plan” 

2.  General  reluctance  to  change 
plans 

7 

Examine 

Other 

Peoples’ 

Perspectives 

The  ability  to  view  and 
inteipret  a  set  of 
circumstances  from  the 
perspectives  of  different 
individuals,  different 
cultures/religions,  and 
different  timeframes 
(historical  perspective). 

Interpret  reports  of 
recent  enemy 
activities  in  area  of 
interest 

Fail  to  accurately  estimate 
enemy  intent 

8 

Decide  When 
To  Seek 
Information 
Based  On 

Value  And 

Cost 

The  ability  to  evaluate  the 
need  for  of  new  information 
in  terms  of  its  cost  in 

Time,  resources,  and  risk. 

Assess  current 
situation 

Tendency  to  ; 

-spend  too  much  time  planning 
and  gathering  information 
-make  quick  decisions  without 
gathering  more  information 
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